<file_sep>+++

title = "C Lecture - 4"
description = "Exercise 48 ~ 51"
+++

Author: <NAME>

All content comes from Zed's [Lecture Repository](https://github.com/zedshaw/learn-c-the-hard-way-lectures.git) and [Libraries Repository](https://github.com/zedshaw/liblcthw).  All credit goes to Zed.



### Exercise 48a A Simple Network Server:

Project Description

The Plan

Start your first long running project:

statserve

The Purpose

You'll get the project started and get a minimum first hack going.

The Requirements

1. Create a simple network server that accepts a connection on port 7899 from
   *netclient* or the *nc* command, and echoes back anything you type.
2. You'll need to learn how to bind a port, listen on the socket, and answer it.
   Use your research skills to study how this is done and attempt to implement it
   yourself.

The Requirements

3. The more important part of this project is laying out the project directory
   from the *c-skeleton*, and making sure you can build everything and get it
   working.
4. Don't worry about things like daemons or anything else.  Your server just has
   to run from the command line and keep running.

The Clues

I will now give you some clues:

* USE liblcthw!
* Remember you did a client already, you just need to make a server.
* Do NOT use select! Use fork() for the server.
* Keep it *simple*.  Don't worry about anything other than accepting a connection and closing.
* Stay small, build slowly.

Important References

* Research online for "echo server in C".
* Read man (2) pages for *accept*, *bind*, *listen*, *connect*, *select*, *socket*, and *shutdown*.

Encouragement

This will be *HARD*!  Try it your best, and take it piece by piece.  You can do it, but remember if you give up the next video (48b) will show you the code to my solution and how to solve it.  You can peek there then come back when you're stuck.

### Exercise 48b A Simple Network Server:
.\ex48b\c-skeleton

.\ex48b\c-skeleton\src\dbg.h

```c
#ifndef __dbg_h__
#define __dbg_h__

#include <stdio.h>
#include <errno.h>
#include <string.h>

#ifdef NDEBUG
#define debug(M, ...)
#else
#define debug(M, ...) fprintf(stderr, "DEBUG %s:%d: " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)
#endif

#define clean_errno() (errno == 0 ? "None" : strerror(errno))

#define log_err(M, ...) fprintf(stderr,\
        "[ERROR] (%s:%d: errno: %s) " M "\n", __FILE__, __LINE__,\
        clean_errno(), ##__VA_ARGS__)

#define log_warn(M, ...) fprintf(stderr,\
        "[WARN] (%s:%d: errno: %s) " M "\n",\
        __FILE__, __LINE__, clean_errno(), ##__VA_ARGS__)

#define log_info(M, ...) fprintf(stderr, "[INFO] (%s:%d) " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)

#define check(A, M, ...) if(!(A)) {\
    log_err(M, ##__VA_ARGS__); errno=0; goto error; }

#define sentinel(M, ...)  { log_err(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#define check_mem(A) check((A), "Out of memory.")

#define check_debug(A, M, ...) if(!(A)) { debug(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#endif

```

.\ex48b\c-skeleton\src\libex29.c

```c
#include <stdio.h>
#include <ctype.h>
#include "dbg.h"

int print_a_message(const char *msg)
{
    printf("A STRING: %s\n", msg);

    return 0;
}

int uppercase(const char *msg)
{
    int i = 0;

    // BUG: \0 termination problems
    for(i = 0; msg[i] != '\0'; i++) {
        printf("%c", toupper(msg[i]));
    }

    printf("\n");

    return 0;
}

int lowercase(const char *msg)
{
    int i = 0;

    // BUG: \0 termination problems
    for(i = 0; msg[i] != '\0'; i++) {
        printf("%c", tolower(msg[i]));
    }

    printf("\n");

    return 0;
}

int fail_on_purpose(const char *msg)
{
    return 1;
} 

```

.\ex48b\c-skeleton\tests\libex29_tests.c

```c
#include "minunit.h"
#include <dlfcn.h>

typedef int (*lib_function) (const char *data);
char *lib_file = "build/libYOUR_LIBRARY.so";
void *lib = NULL;

int check_function(const char *func_to_run, const char *data,
        int expected)
{
    lib_function func = dlsym(lib, func_to_run);
    check(func != NULL,
            "Did not find %s function in the library %s: %s", func_to_run,
            lib_file, dlerror());

    int rc = func(data);
    check(rc == expected, "Function %s return %d for data: %s",
            func_to_run, rc, data);

    return 1;
error:
    return 0;
}

char *test_dlopen()
{
    lib = dlopen(lib_file, RTLD_NOW);
    mu_assert(lib != NULL, "Failed to open the library to test.");

    return NULL;
}

char *test_functions()
{
    mu_assert(check_function("print_a_message", "Hello", 0),
            "print_a_message failed.");
    mu_assert(check_function("uppercase", "Hello", 0),
            "uppercase failed.");
    mu_assert(check_function("lowercase", "Hello", 0),
            "lowercase failed.");

    return NULL;
}

char *test_failures()
{
    mu_assert(check_function("fail_on_purpose", "Hello", 1),
            "fail_on_purpose should fail.");

    return NULL;
}

char *test_dlclose()
{
    int rc = dlclose(lib);
    mu_assert(rc == 0, "Failed to close lib.");

    return NULL;
}

char *all_tests()
{
    mu_suite_start();

    mu_run_test(test_dlopen);
    mu_run_test(test_functions);
    mu_run_test(test_failures);
    mu_run_test(test_dlclose);

    return NULL;
}

RUN_TESTS(all_tests);

```

.\ex48b\statserve

.\ex48b\statserve\bin\statserve.c

```c
#include <stdio.h>
#include <lcthw/dbg.h>
#include "statserve.h"
#include "net.h"

int main(int argc, char *argv[])
{
    check(argc == 3, "USAGE: statserve host port");

    const char *host = argv[1];
    const char *port = argv[2];

    check(echo_server(host, port), "Failed to run the echo server.");

    return 0;

error:

    return 1;
}

```

.\ex48b\statserve\src\dbg.h

```c
#ifndef __dbg_h__
#define __dbg_h__

#include <stdio.h>
#include <errno.h>
#include <string.h>

#ifdef NDEBUG
#define debug(M, ...)
#else
#define debug(M, ...) fprintf(stderr, "DEBUG %s:%d: " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)
#endif

#define clean_errno() (errno == 0 ? "None" : strerror(errno))

#define log_err(M, ...) fprintf(stderr,\
        "[ERROR] (%s:%d: errno: %s) " M "\n", __FILE__, __LINE__,\
        clean_errno(), ##__VA_ARGS__)

#define log_warn(M, ...) fprintf(stderr,\
        "[WARN] (%s:%d: errno: %s) " M "\n",\
        __FILE__, __LINE__, clean_errno(), ##__VA_ARGS__)

#define log_info(M, ...) fprintf(stderr, "[INFO] (%s:%d) " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)

#define check(A, M, ...) if(!(A)) {\
    log_err(M, ##__VA_ARGS__); errno=0; goto error; }

#define sentinel(M, ...)  { log_err(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#define check_mem(A) check((A), "Out of memory.")

#define check_debug(A, M, ...) if(!(A)) { debug(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#endif

```

.\ex48b\statserve\src\net.c

```c
#include <stdlib.h>
#include <sys/select.h>
#include <stdio.h>
#include <lcthw/ringbuffer.h>
#include <lcthw/dbg.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <sys/uio.h>
#include <arpa/inet.h>
#include <netdb.h>
#include <unistd.h>
#include <fcntl.h>
#include "net.h"

struct tagbstring NL = bsStatic("\n");
struct tagbstring CRLF = bsStatic("\r\n");

int nonblock(int fd)
{
    int flags = fcntl(fd, F_GETFL, 0);
    check(flags >= 0, "Invalid flags on nonblock.");

    int rc = fcntl(fd, F_SETFL, flags | O_NONBLOCK);
    check(rc == 0, "Can't set nonblocking.");

    return 0;
error:
    return -1;
}

int client_connect(char *host, char *port)
{
    int rc = 0;
    struct addrinfo *addr = NULL;

    rc = getaddrinfo(host, port, NULL, &addr);
    check(rc == 0, "Failed to lookup %s:%s", host, port);

    int sock = socket(AF_INET, SOCK_STREAM, 0);
    check(sock >= 0, "Cannot create a socket.");

    rc = connect(sock, addr->ai_addr, addr->ai_addrlen);
    check(rc == 0, "Connect failed.");

    rc = nonblock(sock);
    check(rc == 0, "Can't set nonblocking.");

    freeaddrinfo(addr);
    return sock;

error:
    freeaddrinfo(addr);
    return -1;
}

int read_some(RingBuffer * buffer, int fd, int is_socket)
{
    int rc = 0;

    if (RingBuffer_available_data(buffer) == 0) {
        buffer->start = buffer->end = 0;
    }

    if (is_socket) {
        rc = recv(fd, RingBuffer_starts_at(buffer),
                RingBuffer_available_space(buffer), 0);
    } else {
        rc = read(fd, RingBuffer_starts_at(buffer),
                RingBuffer_available_space(buffer));
    }

    check(rc >= 0, "Failed to read from fd: %d", fd);

    RingBuffer_commit_write(buffer, rc);

    return rc;

error:
    return -1;
}

int write_some(RingBuffer * buffer, int fd, int is_socket)
{
    int rc = 0;
    bstring data = RingBuffer_get_all(buffer);

    check(data != NULL, "Failed to get from the buffer.");
    check(bfindreplace(data, &NL, &CRLF, 0) == BSTR_OK,
            "Failed to replace NL.");

    if (is_socket) {
        rc = send(fd, bdata(data), blength(data), 0);
    } else {
        rc = write(fd, bdata(data), blength(data));
    }

    check(rc == blength(data), "Failed to write everything to fd: %d.",
            fd);
    bdestroy(data);

    return rc;

error:
    return -1;
}

int attempt_listen(struct addrinfo *info)
{
    int sockfd = -1; // default fail
    int rc = -1;
    int yes = 1;

    check(info != NULL, "Invalid addrinfo.");

    // create a socket with the addrinfo
    sockfd = socket(info->ai_family, info->ai_socktype,
            info->ai_protocol);
    check_debug(sockfd != -1, "Failed to bind to address. Trying more.");

    // set the SO_REUSEADDR option on the socket
    rc = setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(int));
    check_debug(rc == 0, "Failed to set SO_REUSADDR.");

    // attempt to bind to it
    rc = bind(sockfd, info->ai_addr, info->ai_addrlen);
    check_debug(rc == 0, "Failed to find socket.");

    // finally listen with a backlog
    rc = listen(sockfd, BACKLOG);
    check_debug(rc == 0, "Failed to listen to socket.");

    return sockfd;

error:
    return -1;
}

int server_listen(const char *host, const char *port)
{
    int rc = 0;
    int sockfd = -1; // default fail value
    struct addrinfo *info = NULL;
    struct addrinfo *next_p = NULL;
    struct addrinfo addr = {
        .ai_family = AF_UNSPEC,
        .ai_socktype = SOCK_STREAM,
        .ai_flags = AI_PASSIVE
    };

    check(host != NULL, "Invalid host.");
    check(port != NULL, "Invalid port.");

    // get the address info for host and port
    rc = getaddrinfo(NULL, port, &addr, &info);
    check(rc == 0, "Failed to get address info for connect.");

    // cycle through the available list to find one
    for(next_p = info; next_p != NULL; next_p = next_p->ai_next)
    {
        // attempt to listen to each one
        sockfd = attempt_listen(next_p);
        if(sockfd != -1) break;
    }

    // either we found one and were able to listen or nothing.
    check(sockfd != -1, "All possible addresses failed.");

error: //fallthrough
    if(info) freeaddrinfo(info);
    // this gets set by the above to either -1 or valid
    return sockfd;
}

```

.\ex48b\statserve\src\net.h

```c
#ifndef _net_h
#define _net_h

#include <lcthw/ringbuffer.h>

#define BACKLOG 10

int nonblock(int fd);
int client_connect(char *host, char *port);
int read_some(RingBuffer * buffer, int fd, int is_socket);
int write_some(RingBuffer * buffer, int fd, int is_socket);
int server_listen(const char *host, const char *port);

#endif

```

.\ex48b\statserve\src\statserve.c

```c
#include <stdio.h>
#include <ctype.h>
#include <lcthw/dbg.h>
#include <unistd.h>
#include <stdlib.h>
#include <signal.h>
#include <sys/wait.h>
#include "net.h"
#include <netdb.h>

const int RB_SIZE = 1024 * 10;

void handle_sigchild(int sig) {
    sig = 0; // ignore it
    while(waitpid(-1, NULL, WNOHANG) > 0) {
    }
}

void client_handler(int client_fd)
{
    int rc = 0;
    // need a ringbuffer for the input
    RingBuffer *sock_rb = RingBuffer_create(RB_SIZE);

    // read_some in a loop
    while(read_some(sock_rb, client_fd, 1) != -1) {
        // write_it back off the ringbuffer
        if(write_some(sock_rb, client_fd, 1) == -1) {
            debug("Client closed.");
            break;
        }
    }

    // close the socket
    rc = close(client_fd);
    check(rc != -1, "Failed to close the socket.");

error: // fallthrough
    if(sock_rb) RingBuffer_destroy(sock_rb);
    exit(0); // just exit the child process
}

int echo_server(const char *host, const char *port)
{
    int rc = 0;
    struct sockaddr_in client_addr;
    socklen_t sin_size = sizeof(client_addr);
    int server_socket = 0;
    int client_fd = 0;

    struct sigaction sa = {
        .sa_handler = handle_sigchild,
        .sa_flags = SA_RESTART | SA_NOCLDSTOP
    };

    check(host != NULL, "Invalid host.");
    check(port != NULL, "Invalid port.");

    // create a sigaction that handles SIGCHLD
    sigemptyset(&sa.sa_mask);
    rc = sigaction(SIGCHLD, &sa, 0);
    check(rc != -1, "Failed to setup signal handler for child processes.");

    // listen on the given port and host
    server_socket = server_listen(host, port);
    check(server_socket >= 0, "bind to %s:%s failed.", host, port);

    while(1) {
        // accept the connection
        client_fd = accept(server_socket, (struct sockaddr *)&client_addr, &sin_size); 
        check(client_fd >= 0, "Failed to accept connection.");

        debug("Client connected.");

        rc = fork();
        if(rc == 0) {
            // child process
            close(server_socket); // don't need this
            // handle the client
            client_handler(client_fd);
        } else {
            // server process
            close(client_fd); // don't need this
        }
    }

error:  // fallthrough
    return -1;
}

```

.\ex48b\statserve\src\statserve.h

```c
#ifndef _statserve_h
#define _statserve_h

int echo_server(const char *host, const char *port);

#endif

```

.\ex48b\statserve\tests\statserve_tests.c

```c
#include "minunit.h"
#include <dlfcn.h>
#include "statserve.h"

char *test_dummy()
{
    return NULL;
}

char *all_tests()
{
    mu_suite_start();

    mu_run_test(test_dummy);

    return NULL;
}

RUN_TESTS(all_tests);

```

.\ex48b\c-skeleton

.\ex48b\c-skeleton\src\dbg.h

```c
#ifndef __dbg_h__
#define __dbg_h__

#include <stdio.h>
#include <errno.h>
#include <string.h>

#ifdef NDEBUG
#define debug(M, ...)
#else
#define debug(M, ...) fprintf(stderr, "DEBUG %s:%d: " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)
#endif

#define clean_errno() (errno == 0 ? "None" : strerror(errno))

#define log_err(M, ...) fprintf(stderr,\
        "[ERROR] (%s:%d: errno: %s) " M "\n", __FILE__, __LINE__,\
        clean_errno(), ##__VA_ARGS__)

#define log_warn(M, ...) fprintf(stderr,\
        "[WARN] (%s:%d: errno: %s) " M "\n",\
        __FILE__, __LINE__, clean_errno(), ##__VA_ARGS__)

#define log_info(M, ...) fprintf(stderr, "[INFO] (%s:%d) " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)

#define check(A, M, ...) if(!(A)) {\
    log_err(M, ##__VA_ARGS__); errno=0; goto error; }

#define sentinel(M, ...)  { log_err(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#define check_mem(A) check((A), "Out of memory.")

#define check_debug(A, M, ...) if(!(A)) { debug(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#endif

```

.\ex48b\c-skeleton\src\libex29.c

```c
#include <stdio.h>
#include <ctype.h>
#include "dbg.h"

int print_a_message(const char *msg)
{
    printf("A STRING: %s\n", msg);

    return 0;
}

int uppercase(const char *msg)
{
    int i = 0;

    // BUG: \0 termination problems
    for(i = 0; msg[i] != '\0'; i++) {
        printf("%c", toupper(msg[i]));
    }

    printf("\n");

    return 0;
}

int lowercase(const char *msg)
{
    int i = 0;

    // BUG: \0 termination problems
    for(i = 0; msg[i] != '\0'; i++) {
        printf("%c", tolower(msg[i]));
    }

    printf("\n");

    return 0;
}

int fail_on_purpose(const char *msg)
{
    return 1;
} 

```

.\ex48b\c-skeleton\tests\libex29_tests.c

```c
#include "minunit.h"
#include <dlfcn.h>

typedef int (*lib_function) (const char *data);
char *lib_file = "build/libYOUR_LIBRARY.so";
void *lib = NULL;

int check_function(const char *func_to_run, const char *data,
        int expected)
{
    lib_function func = dlsym(lib, func_to_run);
    check(func != NULL,
            "Did not find %s function in the library %s: %s", func_to_run,
            lib_file, dlerror());

    int rc = func(data);
    check(rc == expected, "Function %s return %d for data: %s",
            func_to_run, rc, data);

    return 1;
error:
    return 0;
}

char *test_dlopen()
{
    lib = dlopen(lib_file, RTLD_NOW);
    mu_assert(lib != NULL, "Failed to open the library to test.");

    return NULL;
}

char *test_functions()
{
    mu_assert(check_function("print_a_message", "Hello", 0),
            "print_a_message failed.");
    mu_assert(check_function("uppercase", "Hello", 0),
            "uppercase failed.");
    mu_assert(check_function("lowercase", "Hello", 0),
            "lowercase failed.");

    return NULL;
}

char *test_failures()
{
    mu_assert(check_function("fail_on_purpose", "Hello", 1),
            "fail_on_purpose should fail.");

    return NULL;
}

char *test_dlclose()
{
    int rc = dlclose(lib);
    mu_assert(rc == 0, "Failed to close lib.");

    return NULL;
}

char *all_tests()
{
    mu_suite_start();

    mu_run_test(test_dlopen);
    mu_run_test(test_functions);
    mu_run_test(test_failures);
    mu_run_test(test_dlclose);

    return NULL;
}

RUN_TESTS(all_tests);

```

.\ex48b\statserve

.\ex48b\statserve\bin\statserve.c

```c
#include <stdio.h>
#include <lcthw/dbg.h>
#include "statserve.h"
#include "net.h"

int main(int argc, char *argv[])
{
    check(argc == 3, "USAGE: statserve host port");

    const char *host = argv[1];
    const char *port = argv[2];

    check(echo_server(host, port), "Failed to run the echo server.");

    return 0;

error:

    return 1;
}

```

.\ex48b\statserve\src\dbg.h

```c
#ifndef __dbg_h__
#define __dbg_h__

#include <stdio.h>
#include <errno.h>
#include <string.h>

#ifdef NDEBUG
#define debug(M, ...)
#else
#define debug(M, ...) fprintf(stderr, "DEBUG %s:%d: " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)
#endif

#define clean_errno() (errno == 0 ? "None" : strerror(errno))

#define log_err(M, ...) fprintf(stderr,\
        "[ERROR] (%s:%d: errno: %s) " M "\n", __FILE__, __LINE__,\
        clean_errno(), ##__VA_ARGS__)

#define log_warn(M, ...) fprintf(stderr,\
        "[WARN] (%s:%d: errno: %s) " M "\n",\
        __FILE__, __LINE__, clean_errno(), ##__VA_ARGS__)

#define log_info(M, ...) fprintf(stderr, "[INFO] (%s:%d) " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)

#define check(A, M, ...) if(!(A)) {\
    log_err(M, ##__VA_ARGS__); errno=0; goto error; }

#define sentinel(M, ...)  { log_err(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#define check_mem(A) check((A), "Out of memory.")

#define check_debug(A, M, ...) if(!(A)) { debug(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#endif

```

.\ex48b\statserve\src\net.c

```c
#include <stdlib.h>
#include <sys/select.h>
#include <stdio.h>
#include <lcthw/ringbuffer.h>
#include <lcthw/dbg.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <sys/uio.h>
#include <arpa/inet.h>
#include <netdb.h>
#include <unistd.h>
#include <fcntl.h>
#include "net.h"

struct tagbstring NL = bsStatic("\n");
struct tagbstring CRLF = bsStatic("\r\n");

int nonblock(int fd)
{
    int flags = fcntl(fd, F_GETFL, 0);
    check(flags >= 0, "Invalid flags on nonblock.");

    int rc = fcntl(fd, F_SETFL, flags | O_NONBLOCK);
    check(rc == 0, "Can't set nonblocking.");

    return 0;
error:
    return -1;
}

int client_connect(char *host, char *port)
{
    int rc = 0;
    struct addrinfo *addr = NULL;

    rc = getaddrinfo(host, port, NULL, &addr);
    check(rc == 0, "Failed to lookup %s:%s", host, port);

    int sock = socket(AF_INET, SOCK_STREAM, 0);
    check(sock >= 0, "Cannot create a socket.");

    rc = connect(sock, addr->ai_addr, addr->ai_addrlen);
    check(rc == 0, "Connect failed.");

    rc = nonblock(sock);
    check(rc == 0, "Can't set nonblocking.");

    freeaddrinfo(addr);
    return sock;

error:
    freeaddrinfo(addr);
    return -1;
}

int read_some(RingBuffer * buffer, int fd, int is_socket)
{
    int rc = 0;

    if (RingBuffer_available_data(buffer) == 0) {
        buffer->start = buffer->end = 0;
    }

    if (is_socket) {
        rc = recv(fd, RingBuffer_starts_at(buffer),
                RingBuffer_available_space(buffer), 0);
    } else {
        rc = read(fd, RingBuffer_starts_at(buffer),
                RingBuffer_available_space(buffer));
    }

    check(rc >= 0, "Failed to read from fd: %d", fd);

    RingBuffer_commit_write(buffer, rc);

    return rc;

error:
    return -1;
}

int write_some(RingBuffer * buffer, int fd, int is_socket)
{
    int rc = 0;
    bstring data = RingBuffer_get_all(buffer);

    check(data != NULL, "Failed to get from the buffer.");
    check(bfindreplace(data, &NL, &CRLF, 0) == BSTR_OK,
            "Failed to replace NL.");

    if (is_socket) {
        rc = send(fd, bdata(data), blength(data), 0);
    } else {
        rc = write(fd, bdata(data), blength(data));
    }

    check(rc == blength(data), "Failed to write everything to fd: %d.",
            fd);
    bdestroy(data);

    return rc;

error:
    return -1;
}

int attempt_listen(struct addrinfo *info)
{
    int sockfd = -1; // default fail
    int rc = -1;
    int yes = 1;

    check(info != NULL, "Invalid addrinfo.");

    // create a socket with the addrinfo
    sockfd = socket(info->ai_family, info->ai_socktype,
            info->ai_protocol);
    check_debug(sockfd != -1, "Failed to bind to address. Trying more.");

    // set the SO_REUSEADDR option on the socket
    rc = setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(int));
    check_debug(rc == 0, "Failed to set SO_REUSADDR.");

    // attempt to bind to it
    rc = bind(sockfd, info->ai_addr, info->ai_addrlen);
    check_debug(rc == 0, "Failed to find socket.");

    // finally listen with a backlog
    rc = listen(sockfd, BACKLOG);
    check_debug(rc == 0, "Failed to listen to socket.");

    return sockfd;

error:
    return -1;
}

int server_listen(const char *host, const char *port)
{
    int rc = 0;
    int sockfd = -1; // default fail value
    struct addrinfo *info = NULL;
    struct addrinfo *next_p = NULL;
    struct addrinfo addr = {
        .ai_family = AF_UNSPEC,
        .ai_socktype = SOCK_STREAM,
        .ai_flags = AI_PASSIVE
    };

    check(host != NULL, "Invalid host.");
    check(port != NULL, "Invalid port.");

    // get the address info for host and port
    rc = getaddrinfo(NULL, port, &addr, &info);
    check(rc == 0, "Failed to get address info for connect.");

    // cycle through the available list to find one
    for(next_p = info; next_p != NULL; next_p = next_p->ai_next)
    {
        // attempt to listen to each one
        sockfd = attempt_listen(next_p);
        if(sockfd != -1) break;
    }

    // either we found one and were able to listen or nothing.
    check(sockfd != -1, "All possible addresses failed.");

error: //fallthrough
    if(info) freeaddrinfo(info);
    // this gets set by the above to either -1 or valid
    return sockfd;
}

```

.\ex48b\statserve\src\net.h

```c
#ifndef _net_h
#define _net_h

#include <lcthw/ringbuffer.h>

#define BACKLOG 10

int nonblock(int fd);
int client_connect(char *host, char *port);
int read_some(RingBuffer * buffer, int fd, int is_socket);
int write_some(RingBuffer * buffer, int fd, int is_socket);
int server_listen(const char *host, const char *port);

#endif

```

.\ex48b\statserve\src\statserve.c

```c
#include <stdio.h>
#include <ctype.h>
#include <lcthw/dbg.h>
#include <unistd.h>
#include <stdlib.h>
#include <signal.h>
#include <sys/wait.h>
#include "net.h"
#include <netdb.h>

const int RB_SIZE = 1024 * 10;

void handle_sigchild(int sig) {
    sig = 0; // ignore it
    while(waitpid(-1, NULL, WNOHANG) > 0) {
    }
}

void client_handler(int client_fd)
{
    int rc = 0;
    // need a ringbuffer for the input
    RingBuffer *sock_rb = RingBuffer_create(RB_SIZE);

    // read_some in a loop
    while(read_some(sock_rb, client_fd, 1) != -1) {
        // write_it back off the ringbuffer
        if(write_some(sock_rb, client_fd, 1) == -1) {
            debug("Client closed.");
            break;
        }
    }

    // close the socket
    rc = close(client_fd);
    check(rc != -1, "Failed to close the socket.");

error: // fallthrough
    if(sock_rb) RingBuffer_destroy(sock_rb);
    exit(0); // just exit the child process
}

int echo_server(const char *host, const char *port)
{
    int rc = 0;
    struct sockaddr_in client_addr;
    socklen_t sin_size = sizeof(client_addr);
    int server_socket = 0;
    int client_fd = 0;

    struct sigaction sa = {
        .sa_handler = handle_sigchild,
        .sa_flags = SA_RESTART | SA_NOCLDSTOP
    };

    check(host != NULL, "Invalid host.");
    check(port != NULL, "Invalid port.");

    // create a sigaction that handles SIGCHLD
    sigemptyset(&sa.sa_mask);
    rc = sigaction(SIGCHLD, &sa, 0);
    check(rc != -1, "Failed to setup signal handler for child processes.");

    // listen on the given port and host
    server_socket = server_listen(host, port);
    check(server_socket >= 0, "bind to %s:%s failed.", host, port);

    while(1) {
        // accept the connection
        client_fd = accept(server_socket, (struct sockaddr *)&client_addr, &sin_size); 
        check(client_fd >= 0, "Failed to accept connection.");

        debug("Client connected.");

        rc = fork();
        if(rc == 0) {
            // child process
            close(server_socket); // don't need this
            // handle the client
            client_handler(client_fd);
        } else {
            // server process
            close(client_fd); // don't need this
        }
    }

error:  // fallthrough
    return -1;
}

```

.\ex48b\statserve\src\statserve.h

```c
#ifndef _statserve_h
#define _statserve_h

int echo_server(const char *host, const char *port);

#endif

```

.\ex48b\statserve\tests\statserve_tests.c

```c
#include "minunit.h"
#include <dlfcn.h>
#include "statserve.h"

char *test_dummy()
{
    return NULL;
}

char *all_tests()
{
    mu_suite_start();

    mu_run_test(test_dummy);

    return NULL;
}

RUN_TESTS(all_tests);

```

Solution

The Plan

Show you how I solved the *statserve* project.

The Purpose

Watch me solve the first project quickly, then review the code.

The Setup

First I need to install liblcthw since I'll be using that.

Then I make the project skeleton and get something, anything going.

The Server

Then I just get it accepting a connection.

The Echo

Then I decided to just make it echo back what I type.

The Final Code

### Exercise 49a A Statistics Server

Project Description

The Plan

Make the *statsserver* do something using a simple protocol.

The Purpose

Learn the first steps in creating a server that answers a protocol.

The Requirements

Create this protocol:

    create Create a new statistic.
    mean   Get the current mean of a statistic.
    sample Add a new sample to a statistics.
    dump   Get all of the elements of a statistic (sum, sumsq, n, min, and max).

The Requirements

1. You'll need to allow people to name these statistics, which means using one of the map style data structures to map names to ``Stats`` structs.
2. You'll need to add the ``CRUD`` standard operations for each name.  CRUD stands for create read update delete.  Currently, the list of commands above has create, mean, and dump for reading; and sample for updating.  You need a delete command now.
3. Make the protocol *strict*! Abort any client that makes any mistakes in protocols.

Strict Protocol

Once again, in case you missed it, be ruthless!

Abort all deviant clients.

Pause!

I'm going to give you clues to solve this, so if you want to try on your own pause now!

The Clues

* Create the data structures first for holding the information for each of these commands.
* Then write a protocol parser to handle it and fill in the data.
* Then pass that data to a function that knows how to do that command.
* You can just store the stats in a Hashmap, BSTree, or TSTree for now.
* KEEP IT SIMPLE!

Important References

* You'll want to refer to the bstring documentation as much as possible to know what functions to use.

Encouragement

* Remember that this is *supposed* to be hard.
* You are *supposed* to struggle with this.
* This could take you a while, but keep up the struggle, do it bit by bit, and test little pieces as you go.
* Automate your tests!

### Exercise 49b A Statistics Server:
.\ex49b\statserve

.\ex49b\statserve\bin\statserve.c

```c
#include <stdio.h>
#include <lcthw/dbg.h>
#include "statserve.h"
#include "net.h"

int main(int argc, char *argv[])
{
    check(argc == 3, "USAGE: statserve host port");

    const char *host = argv[1];
    const char *port = argv[2];

    check(echo_server(host, port), "Failed to run the echo server.");

    return 0;

error:

    return 1;
}

```

.\ex49b\statserve\src\dbg.h

```c
#ifndef __dbg_h__
#define __dbg_h__

#include <stdio.h>
#include <errno.h>
#include <string.h>

#ifdef NDEBUG
#define debug(M, ...)
#else
#define debug(M, ...) fprintf(stderr, "DEBUG %s:%d: " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)
#endif

#define clean_errno() (errno == 0 ? "None" : strerror(errno))

#define log_err(M, ...) fprintf(stderr,\
        "[ERROR] (%s:%d: errno: %s) " M "\n", __FILE__, __LINE__,\
        clean_errno(), ##__VA_ARGS__)

#define log_warn(M, ...) fprintf(stderr,\
        "[WARN] (%s:%d: errno: %s) " M "\n",\
        __FILE__, __LINE__, clean_errno(), ##__VA_ARGS__)

#define log_info(M, ...) fprintf(stderr, "[INFO] (%s:%d) " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)

#define check(A, M, ...) if(!(A)) {\
    log_err(M, ##__VA_ARGS__); errno=0; goto error; }

#define sentinel(M, ...)  { log_err(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#define check_mem(A) check((A), "Out of memory.")

#define check_debug(A, M, ...) if(!(A)) { debug(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#endif

```

.\ex49b\statserve\src\net.c

```c
#include <stdlib.h>
#include <sys/select.h>
#include <stdio.h>
#include <lcthw/ringbuffer.h>
#include <lcthw/dbg.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <sys/uio.h>
#include <arpa/inet.h>
#include <netdb.h>
#include <unistd.h>
#include <fcntl.h>
#include "net.h"

struct tagbstring NL = bsStatic("\n");
struct tagbstring CRLF = bsStatic("\r\n");

int nonblock(int fd)
{
    int flags = fcntl(fd, F_GETFL, 0);
    check(flags >= 0, "Invalid flags on nonblock.");

    int rc = fcntl(fd, F_SETFL, flags | O_NONBLOCK);
    check(rc == 0, "Can't set nonblocking.");

    return 0;
error:
    return -1;
}

int client_connect(char *host, char *port)
{
    int rc = 0;
    struct addrinfo *addr = NULL;

    rc = getaddrinfo(host, port, NULL, &addr);
    check(rc == 0, "Failed to lookup %s:%s", host, port);

    int sock = socket(AF_INET, SOCK_STREAM, 0);
    check(sock >= 0, "Cannot create a socket.");

    rc = connect(sock, addr->ai_addr, addr->ai_addrlen);
    check(rc == 0, "Connect failed.");

    rc = nonblock(sock);
    check(rc == 0, "Can't set nonblocking.");

    freeaddrinfo(addr);
    return sock;

error:
    freeaddrinfo(addr);
    return -1;
}

int read_some(RingBuffer * buffer, int fd, int is_socket)
{
    int rc = 0;

    if (RingBuffer_available_data(buffer) == 0) {
        buffer->start = buffer->end = 0;
    }

    if (is_socket) {
        rc = recv(fd, RingBuffer_starts_at(buffer),
                RingBuffer_available_space(buffer), 0);
    } else {
        rc = read(fd, RingBuffer_starts_at(buffer),
                RingBuffer_available_space(buffer));
    }

    check(rc >= 0, "Failed to read from fd: %d", fd);

    RingBuffer_commit_write(buffer, rc);

    return rc;

error:
    return -1;
}

int write_some(RingBuffer * buffer, int fd, int is_socket)
{
    int rc = 0;
    bstring data = RingBuffer_get_all(buffer);

    check(data != NULL, "Failed to get from the buffer.");
    check(bfindreplace(data, &NL, &CRLF, 0) == BSTR_OK,
            "Failed to replace NL.");

    if (is_socket) {
        rc = send(fd, bdata(data), blength(data), 0);
    } else {
        rc = write(fd, bdata(data), blength(data));
    }

    check(rc == blength(data), "Failed to write everything to fd: %d.",
            fd);
    bdestroy(data);

    return rc;

error:
    return -1;
}

int attempt_listen(struct addrinfo *info)
{
    int sockfd = -1; // default fail
    int rc = -1;
    int yes = 1;

    check(info != NULL, "Invalid addrinfo.");

    // create a socket with the addrinfo
    sockfd = socket(info->ai_family, info->ai_socktype,
            info->ai_protocol);
    check_debug(sockfd != -1, "Failed to bind to address. Trying more.");

    // set the SO_REUSEADDR option on the socket
    rc = setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(int));
    check_debug(rc == 0, "Failed to set SO_REUSADDR.");

    // attempt to bind to it
    rc = bind(sockfd, info->ai_addr, info->ai_addrlen);
    check_debug(rc == 0, "Failed to find socket.");

    // finally listen with a backlog
    rc = listen(sockfd, BACKLOG);
    check_debug(rc == 0, "Failed to listen to socket.");

    return sockfd;

error:
    return -1;
}

int server_listen(const char *host, const char *port)
{
    int rc = 0;
    int sockfd = -1; // default fail value
    struct addrinfo *info = NULL;
    struct addrinfo *next_p = NULL;
    struct addrinfo addr = {
        .ai_family = AF_UNSPEC,
        .ai_socktype = SOCK_STREAM,
        .ai_flags = AI_PASSIVE
    };

    check(host != NULL, "Invalid host.");
    check(port != NULL, "Invalid port.");

    // get the address info for host and port
    rc = getaddrinfo(NULL, port, &addr, &info);
    check(rc == 0, "Failed to get address info for connect.");

    // cycle through the available list to find one
    for(next_p = info; next_p != NULL; next_p = next_p->ai_next)
    {
        // attempt to listen to each one
        sockfd = attempt_listen(next_p);
        if(sockfd != -1) break;
    }

    // either we found one and were able to listen or nothing.
    check(sockfd != -1, "All possible addresses failed.");

error: //fallthrough
    if(info) freeaddrinfo(info);
    // this gets set by the above to either -1 or valid
    return sockfd;
}

bstring read_line(RingBuffer *input, const char line_ending)
{
    int i = 0;
    bstring result = NULL;

    // not super efficient
    // read a character at a time from the ring buffer
    for(i = 0; i < RingBuffer_available_data(input); i++) {
        // if the buffer has line ending
        if(input->buffer[i] == line_ending) {
            // get that much fromt he ring buffer
            result = RingBuffer_gets(input, i);
            check(result, "Failed to get line from RingBuffer");
            // make sure that we got the right amount
            check(RingBuffer_available_data(input) >= 1, 
                    "Not enough data in the RingBuffer after reading line.");
            // and commit it
            RingBuffer_commit_read(input, 1);
            break;
        }
    }

    // notice this will fail in the cases where we get a set of data
    // on the wire that does not have a line ending yet
    return result;
error:
    return NULL;
}

```

.\ex49b\statserve\src\net.h

```c
#ifndef _net_h
#define _net_h

#include <lcthw/ringbuffer.h>

#define BACKLOG 10

int nonblock(int fd);
int client_connect(char *host, char *port);
int read_some(RingBuffer * buffer, int fd, int is_socket);
int write_some(RingBuffer * buffer, int fd, int is_socket);
int server_listen(const char *host, const char *port);
bstring read_line(RingBuffer *input, const char line_ending);

#endif

```

.\ex49b\statserve\src\statserve.c

```c
#include <stdio.h>
#include <ctype.h>
#include <lcthw/dbg.h>
#include <lcthw/hashmap.h>
#include <lcthw/stats.h>
#include <unistd.h>
#include <stdlib.h>
#include <signal.h>
#include <sys/wait.h>
#include "net.h"
#include <netdb.h>

struct tagbstring LINE_SPLIT = bsStatic(" ");
struct tagbstring CREATE = bsStatic("create");
struct tagbstring STDDEV = bsStatic("stddev");
struct tagbstring MEAN = bsStatic("mean");
struct tagbstring SAMPLE = bsStatic("sample");
struct tagbstring DUMP = bsStatic("dump");
struct tagbstring DELETE = bsStatic("delete");
struct tagbstring OK = bsStatic("OK\n");
struct tagbstring ERR = bsStatic("ERR\n");
struct tagbstring DNE = bsStatic("DNE\n");
struct tagbstring EXISTS = bsStatic("EXISTS\n");
const char LINE_ENDING = '\n';

const int RB_SIZE = 1024 * 10;

Hashmap *DATA = NULL;

struct Command;

typedef int (*handler_cb)(struct Command *cmd, RingBuffer *send_rb);

typedef struct Command {
    bstring command;
    bstring name;
    bstring number;
    handler_cb handler;
} Command;

typedef struct Record {
    bstring name;
    Stats *stat;
} Record;

void handle_sigchild(int sig) {
    sig = 0; // ignore it
    while(waitpid(-1, NULL, WNOHANG) > 0) {
    }
}

void send_reply(RingBuffer *send_rb, bstring reply)
{
    RingBuffer_puts(send_rb, reply);
}

int handle_create(Command *cmd, RingBuffer *send_rb)
{
    int rc = 0;

    // if the name is in the DATA map then return exists
    if(Hashmap_get(DATA, cmd->name)) {
        send_reply(send_rb, &EXISTS);
    } else {
        // allocate a recrod
        debug("create: %s %s", bdata(cmd->name), bdata(cmd->number));

        Record *info = calloc(sizeof(Record), 1);
        check_mem(info);

        // set its stat element
        info->stat = Stats_create();
        check_mem(info->stat);

        // set its name element
        info->name = bstrcpy(cmd->name);
        check_mem(info->name);

        // do a first sample
        Stats_sample(info->stat, atof(bdata(cmd->number)));

        // add it to the hashmap
        rc = Hashmap_set(DATA, info->name, info);
        check(rc == 0, "Failed to add data to map.");

        // send an OK
        send_reply(send_rb, &OK);
    }

    return 0;
error:
    return -1;
}

int handle_sample(Command *cmd, RingBuffer *send_rb)
{
    // get the info from the hashmap
    Record *info = Hashmap_get(DATA, cmd->name);

    if(info == NULL) {
        // if it doesn't exist then DNE
        send_reply(send_rb, &DNE);
    } else {
        // else run sample on it, return the mean
        Stats_sample(info->stat, atof(bdata(cmd->number)));
        bstring reply = bformat("%f\n", Stats_mean(info->stat));
        send_reply(send_rb, reply);
        bdestroy(reply);
    }

    return 0;
}

int handle_delete(Command *cmd, RingBuffer *send_rb)
{
    log_info("delete: %s", bdata(cmd->name));
    Record *info = Hashmap_get(DATA, cmd->name);

    if(info == NULL) {
        send_reply(send_rb, &DNE);
    } else {
        Hashmap_delete(DATA, cmd->name);

        free(info->stat);
        bdestroy(info->name);
        free(info);

        send_reply(send_rb, &OK);
    }

    return 0;
}

int handle_mean(Command *cmd, RingBuffer *send_rb)
{
    log_info("mean: %s", bdata(cmd->name));
    Record *info = Hashmap_get(DATA, cmd->name);

    if(info == NULL) {
        send_reply(send_rb, &DNE);
    } else {
        bstring reply = bformat("%f\n", Stats_mean(info->stat));
        send_reply(send_rb, reply);
        bdestroy(reply);
    }

    return 0;
}

int handle_stddev(Command *cmd, RingBuffer *send_rb)
{
    log_info("stddev: %s", bdata(cmd->name));
    Record *info = Hashmap_get(DATA, cmd->name);

    if(info == NULL) {
        send_reply(send_rb, &DNE);
    } else {
        bstring reply = bformat("%f\n", Stats_stddev(info->stat));
        send_reply(send_rb, reply);
        bdestroy(reply);
    }

    return 0;
}

int handle_dump(Command *cmd, RingBuffer *send_rb)
{
    log_info("dump: %s", bdata(cmd->name));
    Record *info = Hashmap_get(DATA, cmd->name);

    if(info == NULL) {
        send_reply(send_rb, &DNE);
    } else {
        bstring reply = bformat("%f %f %f %f %ld %f %f\n",
                Stats_mean(info->stat),
                Stats_stddev(info->stat),
                info->stat->sum,
                info->stat->sumsq,
                info->stat->n,
                info->stat->min,
                info->stat->max);

        send_reply(send_rb, reply);
        bdestroy(reply);
    }

    return 0;
}

int parse_command(struct bstrList *splits, Command *cmd)
{
    // get the command
    cmd->command = splits->entry[0];

    if(biseq(cmd->command, &CREATE)) {
        check(splits->qty == 3, "Failed to parse create: %d", splits->qty);
        cmd->name = splits->entry[1];
        cmd->number = splits->entry[2];
        cmd->handler = handle_create;
    } else if(biseq(cmd->command, &MEAN)) {
        check(splits->qty == 2, "Failed to parse mean: %d", splits->qty);
        cmd->name = splits->entry[1];
        cmd->handler = handle_mean;
    } else if(biseq(cmd->command, &SAMPLE)) {
        check(splits->qty == 3, "Failed to parse sample: %d", splits->qty);
        cmd->name = splits->entry[1];
        cmd->number = splits->entry[2];
        cmd->handler = handle_sample;
    } else if(biseq(cmd->command, &DUMP)) {
        check(splits->qty == 2, "Failed to parse dump: %d", splits->qty);
        cmd->name = splits->entry[1];
        cmd->handler = handle_dump;
    } else if(biseq(cmd->command, &DELETE)) {
        check(splits->qty == 2, "Failed to parse delete: %d", splits->qty);
        cmd->name = splits->entry[1];
        cmd->handler = handle_delete;
    } else if(biseq(cmd->command, &STDDEV)) {
        check(splits->qty == 2, "Failed to parse stddev: %d", splits->qty);
        cmd->name = splits->entry[1];
        cmd->handler = handle_stddev;
    } else {
        sentinel("Failed to parse the command.");
    }

    return 0;
error:
    return -1;
}

int parse_line(bstring data, RingBuffer *send_rb)
{
    int rc = -1;
    Command cmd = {.command = NULL};

    // split data on line boundaries
    struct bstrList *splits = bsplits(data, &LINE_SPLIT);
    check(splits != NULL, "Bad data.");

    // parse it into a command
    rc = parse_command(splits, &cmd);
    check(rc == 0, "Failed to parse command.");

    // call the command handler for that command
    rc = cmd.handler(&cmd, send_rb);

error: // fallthrough
    if(splits) bstrListDestroy(splits);
    return rc;

}

void client_handler(int client_fd)
{
    int rc = 0;
    RingBuffer *recv_rb = RingBuffer_create(RB_SIZE);
    RingBuffer *send_rb = RingBuffer_create(RB_SIZE);

    check_mem(recv_rb);
    check_mem(send_rb);

    // keep reading into the recv buffer and sending on send
    while(read_some(recv_rb, client_fd, 1) != -1) {
        // read a line from the recv_rb
        bstring data = read_line(recv_rb, LINE_ENDING);
        check(data != NULL, "Client closed.");

        // parse it, close on any protocol errors
        rc = parse_line(data, send_rb);
        bdestroy(data); // cleanup here
        check(rc == 0, "Failed to parse user. Closing.");

        // and as long as there's something to send, send it
        if(RingBuffer_available_data(send_rb)) {
            write_some(send_rb, client_fd, 1);
        }
    }

    // close the socket
    rc = close(client_fd);
    check(rc != -1, "Failed to close the socket.");

error: // fallthrough
    if(recv_rb) RingBuffer_destroy(recv_rb);
    if(send_rb) RingBuffer_destroy(send_rb);
    exit(0); // just exit the child process
}

int setup_data_store()
{
    // a more advanced design simply wouldn't use this
    DATA = Hashmap_create(NULL, NULL);
    check_mem(DATA);

    return 0;
error:
    return -1;
}

int echo_server(const char *host, const char *port)
{
    int rc = 0;
    struct sockaddr_in client_addr;
    socklen_t sin_size = sizeof(client_addr);
    int server_socket = 0;
    int client_fd = 0;

    rc = setup_data_store();
    check(rc == 0, "Failed to setup the data store.");

    struct sigaction sa = {
        .sa_handler = handle_sigchild,
        .sa_flags = SA_RESTART | SA_NOCLDSTOP
    };

    check(host != NULL, "Invalid host.");
    check(port != NULL, "Invalid port.");

    // create a sigaction that handles SIGCHLD
    sigemptyset(&sa.sa_mask);
    rc = sigaction(SIGCHLD, &sa, 0);
    check(rc != -1, "Failed to setup signal handler for child processes.");

    // listen on the given port and host
    server_socket = server_listen(host, port);
    check(server_socket >= 0, "bind to %s:%s failed.", host, port);

    while(1) {
        // accept the connection
        client_fd = accept(server_socket, (struct sockaddr *)&client_addr, &sin_size); 
        check(client_fd >= 0, "Failed to accept connection.");

        debug("Client connected.");

        rc = fork();
        if(rc == 0) {
            // child process
            close(server_socket); // don't need this
            // handle the client
            client_handler(client_fd);
        } else {
            // server process
            close(client_fd); // don't need this
        }
    }

error:  // fallthrough
    return -1;
}

```

.\ex49b\statserve\src\statserve.h

```c
#ifndef _statserve_h
#define _statserve_h

#include <lcthw/bstrlib.h>
#include <lcthw/ringbuffer.h>

struct tagbstring OK;

int setup_data_store();

int parse_line(bstring data, RingBuffer *send_rb);

int echo_server(const char *host, const char *port);

#endif

```

.\ex49b\statserve\tests\statserve_tests.c

```c
#include "minunit.h"
#include <dlfcn.h>
#include "statserve.h"
#include <lcthw/bstrlib.h>
#include <lcthw/ringbuffer.h>
#include <assert.h>

typedef struct LineTest {
    char *line;
    bstring result;
    char *description;
} LineTest;

int attempt_line(LineTest test)
{
    int rc = -1;
    bstring result = NULL;

    bstring line = bfromcstr(test.line);
    RingBuffer *send_rb = RingBuffer_create(1024);

    rc = parse_line(line, send_rb);
    check(rc == 0, "Failed to parse line.");

    result = RingBuffer_get_all(send_rb);
    check(result != NULL, "Ring buffer empty.");
    check(biseq(result, test.result), "Got the wrong output: %s expected %s",
            bdata(result), bdata(test.result));

    bdestroy(line);
    RingBuffer_destroy(send_rb);
    return 1; // using 1 for tests
error:

    log_err("Failed to process test %s: got %s", test.line, bdata(result));
    if(line) bdestroy(line);
    if(send_rb) RingBuffer_destroy(send_rb);
    return 0;
}

int run_test_lines(LineTest *tests, int count)
{
    int i = 0;

    for(i = 0; i < count; i++) {
        check(attempt_line(tests[i]), "Failed to run %s", tests[i].description);
    }

    return 1;
error:
    return 0;
}

char *test_create()
{
    LineTest tests[] = {
        {.line = "create /zed 100", .result = &OK, .description = "create zed failed"},
        {.line = "create /joe 100", .result = &OK, .description = "create joe failed"},

    };

    mu_assert(run_test_lines(tests, 2), "Failed to run create tests.");

    return NULL;
}

char *test_sample()
{
    struct tagbstring sample1 = bsStatic("100.000000\n");

    LineTest tests[] = {
        {.line = "sample /zed 100", .result = &sample1, .description = "sample zed failed."}
    };

    mu_assert(run_test_lines(tests, 1), "Failed to run sample tests.");

    return NULL;
}

char *all_tests()
{
    mu_suite_start();

    int rc = setup_data_store();
    mu_assert(rc == 0, "Failed to setup the data store.");

    mu_run_test(test_create);
    mu_run_test(test_sample);

    return NULL;
}

RUN_TESTS(all_tests);

```

.\ex49b\statserve

.\ex49b\statserve\bin\statserve.c

```c
#include <stdio.h>
#include <lcthw/dbg.h>
#include "statserve.h"
#include "net.h"

int main(int argc, char *argv[])
{
    check(argc == 3, "USAGE: statserve host port");

    const char *host = argv[1];
    const char *port = argv[2];

    check(echo_server(host, port), "Failed to run the echo server.");

    return 0;

error:

    return 1;
}

```

.\ex49b\statserve\src\dbg.h

```c
#ifndef __dbg_h__
#define __dbg_h__

#include <stdio.h>
#include <errno.h>
#include <string.h>

#ifdef NDEBUG
#define debug(M, ...)
#else
#define debug(M, ...) fprintf(stderr, "DEBUG %s:%d: " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)
#endif

#define clean_errno() (errno == 0 ? "None" : strerror(errno))

#define log_err(M, ...) fprintf(stderr,\
        "[ERROR] (%s:%d: errno: %s) " M "\n", __FILE__, __LINE__,\
        clean_errno(), ##__VA_ARGS__)

#define log_warn(M, ...) fprintf(stderr,\
        "[WARN] (%s:%d: errno: %s) " M "\n",\
        __FILE__, __LINE__, clean_errno(), ##__VA_ARGS__)

#define log_info(M, ...) fprintf(stderr, "[INFO] (%s:%d) " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)

#define check(A, M, ...) if(!(A)) {\
    log_err(M, ##__VA_ARGS__); errno=0; goto error; }

#define sentinel(M, ...)  { log_err(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#define check_mem(A) check((A), "Out of memory.")

#define check_debug(A, M, ...) if(!(A)) { debug(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#endif

```

.\ex49b\statserve\src\net.c

```c
#include <stdlib.h>
#include <sys/select.h>
#include <stdio.h>
#include <lcthw/ringbuffer.h>
#include <lcthw/dbg.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <sys/uio.h>
#include <arpa/inet.h>
#include <netdb.h>
#include <unistd.h>
#include <fcntl.h>
#include "net.h"

struct tagbstring NL = bsStatic("\n");
struct tagbstring CRLF = bsStatic("\r\n");

int nonblock(int fd)
{
    int flags = fcntl(fd, F_GETFL, 0);
    check(flags >= 0, "Invalid flags on nonblock.");

    int rc = fcntl(fd, F_SETFL, flags | O_NONBLOCK);
    check(rc == 0, "Can't set nonblocking.");

    return 0;
error:
    return -1;
}

int client_connect(char *host, char *port)
{
    int rc = 0;
    struct addrinfo *addr = NULL;

    rc = getaddrinfo(host, port, NULL, &addr);
    check(rc == 0, "Failed to lookup %s:%s", host, port);

    int sock = socket(AF_INET, SOCK_STREAM, 0);
    check(sock >= 0, "Cannot create a socket.");

    rc = connect(sock, addr->ai_addr, addr->ai_addrlen);
    check(rc == 0, "Connect failed.");

    rc = nonblock(sock);
    check(rc == 0, "Can't set nonblocking.");

    freeaddrinfo(addr);
    return sock;

error:
    freeaddrinfo(addr);
    return -1;
}

int read_some(RingBuffer * buffer, int fd, int is_socket)
{
    int rc = 0;

    if (RingBuffer_available_data(buffer) == 0) {
        buffer->start = buffer->end = 0;
    }

    if (is_socket) {
        rc = recv(fd, RingBuffer_starts_at(buffer),
                RingBuffer_available_space(buffer), 0);
    } else {
        rc = read(fd, RingBuffer_starts_at(buffer),
                RingBuffer_available_space(buffer));
    }

    check(rc >= 0, "Failed to read from fd: %d", fd);

    RingBuffer_commit_write(buffer, rc);

    return rc;

error:
    return -1;
}

int write_some(RingBuffer * buffer, int fd, int is_socket)
{
    int rc = 0;
    bstring data = RingBuffer_get_all(buffer);

    check(data != NULL, "Failed to get from the buffer.");
    check(bfindreplace(data, &NL, &CRLF, 0) == BSTR_OK,
            "Failed to replace NL.");

    if (is_socket) {
        rc = send(fd, bdata(data), blength(data), 0);
    } else {
        rc = write(fd, bdata(data), blength(data));
    }

    check(rc == blength(data), "Failed to write everything to fd: %d.",
            fd);
    bdestroy(data);

    return rc;

error:
    return -1;
}

int attempt_listen(struct addrinfo *info)
{
    int sockfd = -1; // default fail
    int rc = -1;
    int yes = 1;

    check(info != NULL, "Invalid addrinfo.");

    // create a socket with the addrinfo
    sockfd = socket(info->ai_family, info->ai_socktype,
            info->ai_protocol);
    check_debug(sockfd != -1, "Failed to bind to address. Trying more.");

    // set the SO_REUSEADDR option on the socket
    rc = setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(int));
    check_debug(rc == 0, "Failed to set SO_REUSADDR.");

    // attempt to bind to it
    rc = bind(sockfd, info->ai_addr, info->ai_addrlen);
    check_debug(rc == 0, "Failed to find socket.");

    // finally listen with a backlog
    rc = listen(sockfd, BACKLOG);
    check_debug(rc == 0, "Failed to listen to socket.");

    return sockfd;

error:
    return -1;
}

int server_listen(const char *host, const char *port)
{
    int rc = 0;
    int sockfd = -1; // default fail value
    struct addrinfo *info = NULL;
    struct addrinfo *next_p = NULL;
    struct addrinfo addr = {
        .ai_family = AF_UNSPEC,
        .ai_socktype = SOCK_STREAM,
        .ai_flags = AI_PASSIVE
    };

    check(host != NULL, "Invalid host.");
    check(port != NULL, "Invalid port.");

    // get the address info for host and port
    rc = getaddrinfo(NULL, port, &addr, &info);
    check(rc == 0, "Failed to get address info for connect.");

    // cycle through the available list to find one
    for(next_p = info; next_p != NULL; next_p = next_p->ai_next)
    {
        // attempt to listen to each one
        sockfd = attempt_listen(next_p);
        if(sockfd != -1) break;
    }

    // either we found one and were able to listen or nothing.
    check(sockfd != -1, "All possible addresses failed.");

error: //fallthrough
    if(info) freeaddrinfo(info);
    // this gets set by the above to either -1 or valid
    return sockfd;
}

bstring read_line(RingBuffer *input, const char line_ending)
{
    int i = 0;
    bstring result = NULL;

    // not super efficient
    // read a character at a time from the ring buffer
    for(i = 0; i < RingBuffer_available_data(input); i++) {
        // if the buffer has line ending
        if(input->buffer[i] == line_ending) {
            // get that much fromt he ring buffer
            result = RingBuffer_gets(input, i);
            check(result, "Failed to get line from RingBuffer");
            // make sure that we got the right amount
            check(RingBuffer_available_data(input) >= 1, 
                    "Not enough data in the RingBuffer after reading line.");
            // and commit it
            RingBuffer_commit_read(input, 1);
            break;
        }
    }

    // notice this will fail in the cases where we get a set of data
    // on the wire that does not have a line ending yet
    return result;
error:
    return NULL;
}

```

.\ex49b\statserve\src\net.h

```c
#ifndef _net_h
#define _net_h

#include <lcthw/ringbuffer.h>

#define BACKLOG 10

int nonblock(int fd);
int client_connect(char *host, char *port);
int read_some(RingBuffer * buffer, int fd, int is_socket);
int write_some(RingBuffer * buffer, int fd, int is_socket);
int server_listen(const char *host, const char *port);
bstring read_line(RingBuffer *input, const char line_ending);

#endif

```

.\ex49b\statserve\src\statserve.c

```c
#include <stdio.h>
#include <ctype.h>
#include <lcthw/dbg.h>
#include <lcthw/hashmap.h>
#include <lcthw/stats.h>
#include <unistd.h>
#include <stdlib.h>
#include <signal.h>
#include <sys/wait.h>
#include "net.h"
#include <netdb.h>

struct tagbstring LINE_SPLIT = bsStatic(" ");
struct tagbstring CREATE = bsStatic("create");
struct tagbstring STDDEV = bsStatic("stddev");
struct tagbstring MEAN = bsStatic("mean");
struct tagbstring SAMPLE = bsStatic("sample");
struct tagbstring DUMP = bsStatic("dump");
struct tagbstring DELETE = bsStatic("delete");
struct tagbstring OK = bsStatic("OK\n");
struct tagbstring ERR = bsStatic("ERR\n");
struct tagbstring DNE = bsStatic("DNE\n");
struct tagbstring EXISTS = bsStatic("EXISTS\n");
const char LINE_ENDING = '\n';

const int RB_SIZE = 1024 * 10;

Hashmap *DATA = NULL;

struct Command;

typedef int (*handler_cb)(struct Command *cmd, RingBuffer *send_rb);

typedef struct Command {
    bstring command;
    bstring name;
    bstring number;
    handler_cb handler;
} Command;

typedef struct Record {
    bstring name;
    Stats *stat;
} Record;

void handle_sigchild(int sig) {
    sig = 0; // ignore it
    while(waitpid(-1, NULL, WNOHANG) > 0) {
    }
}

void send_reply(RingBuffer *send_rb, bstring reply)
{
    RingBuffer_puts(send_rb, reply);
}

int handle_create(Command *cmd, RingBuffer *send_rb)
{
    int rc = 0;

    // if the name is in the DATA map then return exists
    if(Hashmap_get(DATA, cmd->name)) {
        send_reply(send_rb, &EXISTS);
    } else {
        // allocate a recrod
        debug("create: %s %s", bdata(cmd->name), bdata(cmd->number));

        Record *info = calloc(sizeof(Record), 1);
        check_mem(info);

        // set its stat element
        info->stat = Stats_create();
        check_mem(info->stat);

        // set its name element
        info->name = bstrcpy(cmd->name);
        check_mem(info->name);

        // do a first sample
        Stats_sample(info->stat, atof(bdata(cmd->number)));

        // add it to the hashmap
        rc = Hashmap_set(DATA, info->name, info);
        check(rc == 0, "Failed to add data to map.");

        // send an OK
        send_reply(send_rb, &OK);
    }

    return 0;
error:
    return -1;
}

int handle_sample(Command *cmd, RingBuffer *send_rb)
{
    // get the info from the hashmap
    Record *info = Hashmap_get(DATA, cmd->name);

    if(info == NULL) {
        // if it doesn't exist then DNE
        send_reply(send_rb, &DNE);
    } else {
        // else run sample on it, return the mean
        Stats_sample(info->stat, atof(bdata(cmd->number)));
        bstring reply = bformat("%f\n", Stats_mean(info->stat));
        send_reply(send_rb, reply);
        bdestroy(reply);
    }

    return 0;
}

int handle_delete(Command *cmd, RingBuffer *send_rb)
{
    log_info("delete: %s", bdata(cmd->name));
    Record *info = Hashmap_get(DATA, cmd->name);

    if(info == NULL) {
        send_reply(send_rb, &DNE);
    } else {
        Hashmap_delete(DATA, cmd->name);

        free(info->stat);
        bdestroy(info->name);
        free(info);

        send_reply(send_rb, &OK);
    }

    return 0;
}

int handle_mean(Command *cmd, RingBuffer *send_rb)
{
    log_info("mean: %s", bdata(cmd->name));
    Record *info = Hashmap_get(DATA, cmd->name);

    if(info == NULL) {
        send_reply(send_rb, &DNE);
    } else {
        bstring reply = bformat("%f\n", Stats_mean(info->stat));
        send_reply(send_rb, reply);
        bdestroy(reply);
    }

    return 0;
}

int handle_stddev(Command *cmd, RingBuffer *send_rb)
{
    log_info("stddev: %s", bdata(cmd->name));
    Record *info = Hashmap_get(DATA, cmd->name);

    if(info == NULL) {
        send_reply(send_rb, &DNE);
    } else {
        bstring reply = bformat("%f\n", Stats_stddev(info->stat));
        send_reply(send_rb, reply);
        bdestroy(reply);
    }

    return 0;
}

int handle_dump(Command *cmd, RingBuffer *send_rb)
{
    log_info("dump: %s", bdata(cmd->name));
    Record *info = Hashmap_get(DATA, cmd->name);

    if(info == NULL) {
        send_reply(send_rb, &DNE);
    } else {
        bstring reply = bformat("%f %f %f %f %ld %f %f\n",
                Stats_mean(info->stat),
                Stats_stddev(info->stat),
                info->stat->sum,
                info->stat->sumsq,
                info->stat->n,
                info->stat->min,
                info->stat->max);

        send_reply(send_rb, reply);
        bdestroy(reply);
    }

    return 0;
}

int parse_command(struct bstrList *splits, Command *cmd)
{
    // get the command
    cmd->command = splits->entry[0];

    if(biseq(cmd->command, &CREATE)) {
        check(splits->qty == 3, "Failed to parse create: %d", splits->qty);
        cmd->name = splits->entry[1];
        cmd->number = splits->entry[2];
        cmd->handler = handle_create;
    } else if(biseq(cmd->command, &MEAN)) {
        check(splits->qty == 2, "Failed to parse mean: %d", splits->qty);
        cmd->name = splits->entry[1];
        cmd->handler = handle_mean;
    } else if(biseq(cmd->command, &SAMPLE)) {
        check(splits->qty == 3, "Failed to parse sample: %d", splits->qty);
        cmd->name = splits->entry[1];
        cmd->number = splits->entry[2];
        cmd->handler = handle_sample;
    } else if(biseq(cmd->command, &DUMP)) {
        check(splits->qty == 2, "Failed to parse dump: %d", splits->qty);
        cmd->name = splits->entry[1];
        cmd->handler = handle_dump;
    } else if(biseq(cmd->command, &DELETE)) {
        check(splits->qty == 2, "Failed to parse delete: %d", splits->qty);
        cmd->name = splits->entry[1];
        cmd->handler = handle_delete;
    } else if(biseq(cmd->command, &STDDEV)) {
        check(splits->qty == 2, "Failed to parse stddev: %d", splits->qty);
        cmd->name = splits->entry[1];
        cmd->handler = handle_stddev;
    } else {
        sentinel("Failed to parse the command.");
    }

    return 0;
error:
    return -1;
}

int parse_line(bstring data, RingBuffer *send_rb)
{
    int rc = -1;
    Command cmd = {.command = NULL};

    // split data on line boundaries
    struct bstrList *splits = bsplits(data, &LINE_SPLIT);
    check(splits != NULL, "Bad data.");

    // parse it into a command
    rc = parse_command(splits, &cmd);
    check(rc == 0, "Failed to parse command.");

    // call the command handler for that command
    rc = cmd.handler(&cmd, send_rb);

error: // fallthrough
    if(splits) bstrListDestroy(splits);
    return rc;

}

void client_handler(int client_fd)
{
    int rc = 0;
    RingBuffer *recv_rb = RingBuffer_create(RB_SIZE);
    RingBuffer *send_rb = RingBuffer_create(RB_SIZE);

    check_mem(recv_rb);
    check_mem(send_rb);

    // keep reading into the recv buffer and sending on send
    while(read_some(recv_rb, client_fd, 1) != -1) {
        // read a line from the recv_rb
        bstring data = read_line(recv_rb, LINE_ENDING);
        check(data != NULL, "Client closed.");

        // parse it, close on any protocol errors
        rc = parse_line(data, send_rb);
        bdestroy(data); // cleanup here
        check(rc == 0, "Failed to parse user. Closing.");

        // and as long as there's something to send, send it
        if(RingBuffer_available_data(send_rb)) {
            write_some(send_rb, client_fd, 1);
        }
    }

    // close the socket
    rc = close(client_fd);
    check(rc != -1, "Failed to close the socket.");

error: // fallthrough
    if(recv_rb) RingBuffer_destroy(recv_rb);
    if(send_rb) RingBuffer_destroy(send_rb);
    exit(0); // just exit the child process
}

int setup_data_store()
{
    // a more advanced design simply wouldn't use this
    DATA = Hashmap_create(NULL, NULL);
    check_mem(DATA);

    return 0;
error:
    return -1;
}

int echo_server(const char *host, const char *port)
{
    int rc = 0;
    struct sockaddr_in client_addr;
    socklen_t sin_size = sizeof(client_addr);
    int server_socket = 0;
    int client_fd = 0;

    rc = setup_data_store();
    check(rc == 0, "Failed to setup the data store.");

    struct sigaction sa = {
        .sa_handler = handle_sigchild,
        .sa_flags = SA_RESTART | SA_NOCLDSTOP
    };

    check(host != NULL, "Invalid host.");
    check(port != NULL, "Invalid port.");

    // create a sigaction that handles SIGCHLD
    sigemptyset(&sa.sa_mask);
    rc = sigaction(SIGCHLD, &sa, 0);
    check(rc != -1, "Failed to setup signal handler for child processes.");

    // listen on the given port and host
    server_socket = server_listen(host, port);
    check(server_socket >= 0, "bind to %s:%s failed.", host, port);

    while(1) {
        // accept the connection
        client_fd = accept(server_socket, (struct sockaddr *)&client_addr, &sin_size); 
        check(client_fd >= 0, "Failed to accept connection.");

        debug("Client connected.");

        rc = fork();
        if(rc == 0) {
            // child process
            close(server_socket); // don't need this
            // handle the client
            client_handler(client_fd);
        } else {
            // server process
            close(client_fd); // don't need this
        }
    }

error:  // fallthrough
    return -1;
}

```

.\ex49b\statserve\src\statserve.h

```c
#ifndef _statserve_h
#define _statserve_h

#include <lcthw/bstrlib.h>
#include <lcthw/ringbuffer.h>

struct tagbstring OK;

int setup_data_store();

int parse_line(bstring data, RingBuffer *send_rb);

int echo_server(const char *host, const char *port);

#endif

```

.\ex49b\statserve\tests\statserve_tests.c

```c
#include "minunit.h"
#include <dlfcn.h>
#include "statserve.h"
#include <lcthw/bstrlib.h>
#include <lcthw/ringbuffer.h>
#include <assert.h>

typedef struct LineTest {
    char *line;
    bstring result;
    char *description;
} LineTest;

int attempt_line(LineTest test)
{
    int rc = -1;
    bstring result = NULL;

    bstring line = bfromcstr(test.line);
    RingBuffer *send_rb = RingBuffer_create(1024);

    rc = parse_line(line, send_rb);
    check(rc == 0, "Failed to parse line.");

    result = RingBuffer_get_all(send_rb);
    check(result != NULL, "Ring buffer empty.");
    check(biseq(result, test.result), "Got the wrong output: %s expected %s",
            bdata(result), bdata(test.result));

    bdestroy(line);
    RingBuffer_destroy(send_rb);
    return 1; // using 1 for tests
error:

    log_err("Failed to process test %s: got %s", test.line, bdata(result));
    if(line) bdestroy(line);
    if(send_rb) RingBuffer_destroy(send_rb);
    return 0;
}

int run_test_lines(LineTest *tests, int count)
{
    int i = 0;

    for(i = 0; i < count; i++) {
        check(attempt_line(tests[i]), "Failed to run %s", tests[i].description);
    }

    return 1;
error:
    return 0;
}

char *test_create()
{
    LineTest tests[] = {
        {.line = "create /zed 100", .result = &OK, .description = "create zed failed"},
        {.line = "create /joe 100", .result = &OK, .description = "create joe failed"},

    };

    mu_assert(run_test_lines(tests, 2), "Failed to run create tests.");

    return NULL;
}

char *test_sample()
{
    struct tagbstring sample1 = bsStatic("100.000000\n");

    LineTest tests[] = {
        {.line = "sample /zed 100", .result = &sample1, .description = "sample zed failed."}
    };

    mu_assert(run_test_lines(tests, 1), "Failed to run sample tests.");

    return NULL;
}

char *all_tests()
{
    mu_suite_start();

    int rc = setup_data_store();
    mu_assert(rc == 0, "Failed to setup the data store.");

    mu_run_test(test_create);
    mu_run_test(test_sample);

    return NULL;
}

RUN_TESTS(all_tests);

```

Solution

The Plan

I'll show you how I implemented the protocol in the smallest code possible.

I won't implement all of the CRUD operations, so you can go look at the 
git repo for this project to see a full implementation.

The Setup

First I setup the data, then the protocol parser, then the handlers.

The Protocol

    create Create a new statistic.
    mean   Get the current mean of a statistic.
    sample Add a new sample to a statistics.
    dump   Get all of the elements of a statistic (sum, sumsq, n, min, and max).Final Code

The Command Structure

    typedef struct Command {
        bstring command;
        bstring name;
        bstring number;
        handler_cb handler;
    } Command;

The Storage Record

    typedef struct Record {
        bstring name;
        Stats *stat;
    } Record;

The Design

* Accept a connection
* Parse the line into the Command
* Run a handler function to process it
* Temporarily store into a Hashmap

Final Thoughts

The last thing I would do is add better tests and round out the protocol with CRUD operations.

### Exercise 50a Routing The Statistics

Project Description

The Plan

You are now given vague instructions and have to "solve" as best you can.

The Purpose

To give you freedom to be creative, and also taste a real project with vague
specifications.

Many times all you get is a single sentence in a bug tracker. Oh well.

The Requirements

Allow people to work with statistics at arbitrary URLs in the server.
You get to define what that means, but think "web application".

Pause!

Try to solve it on your own then continue.

The Clues

Answer these questions:

1. What happens when I have a statistics "under" another, as in /age/northamerica/ is under /age/.
2. Could you do the summary statistics we talked about?  A mean of means and mean of standard deviations that are rolled up the tree?
3. What data structures do you need?  Starting with data is key here too. Data data data.
4. Are your tests good enough?  Before you start you might want to get good tests that use the protocol.

Important References

* Definitely look at the statistics code you built in liblcthw if you do the summary statistics.

Encouragement

This is hard, as I've said all along, however it is all doable. It's simply a matter of breaking the problems down and tackling each little piece.

### Exercise 50b Routing the Statistics

Solution

The Plan

Show you how I solved the problem of routing the names of statistics as URLs.

The Setup

1. First thing I did was make sure my tests were really good.
2. Then I designed the data structures I'd need.
3. Then I did the work to make them functions.
4. The protocol shouldn't need to change.

"URLs"

I'll define paths as simply names separated by /.

Real URLs are way more complex than that.

Data Structure

I just added:

    struct bstrList *path;

To the Command struct to hold paths.

URL Meaning

Kind of weird, but:

    Deepest part of URL is "parent", this is the main stat.
    Children are next segments up, and are mean-of-mean stats.

New Processing

1. Change to a loop over all paths with a "scan path" function.
2. Add optional path parameter to handlers.
3. Parse the path in *parse\_command* to set path in Commands.
4. In sample and create, change root processing vs. child processing.
5. Move *send\_reply* over to *net.c* instead.

Test First Path Parsing

I'll write a small test for just the *scan\_paths* part first.

Then wire that in and use the existing tests to confirm the old code
works.

The Code

.\ex50b\statserve

.\ex50b\statserve\bin\statserve.c

```c
#include <stdio.h>
#include <lcthw/dbg.h>
#include "statserve.h"
#include "net.h"

int main(int argc, char *argv[])
{
    check(argc == 3, "USAGE: statserve host port");

    const char *host = argv[1];
    const char *port = argv[2];

    check(echo_server(host, port), "Failed to run the echo server.");

    return 0;

error:

    return 1;
}

```

.\ex50b\statserve\src\dbg.h

```c
#ifndef __dbg_h__
#define __dbg_h__

#include <stdio.h>
#include <errno.h>
#include <string.h>

#ifdef NDEBUG
#define debug(M, ...)
#else
#define debug(M, ...) fprintf(stderr, "DEBUG %s:%d: " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)
#endif

#define clean_errno() (errno == 0 ? "None" : strerror(errno))

#define log_err(M, ...) fprintf(stderr,\
        "[ERROR] (%s:%d: errno: %s) " M "\n", __FILE__, __LINE__,\
        clean_errno(), ##__VA_ARGS__)

#define log_warn(M, ...) fprintf(stderr,\
        "[WARN] (%s:%d: errno: %s) " M "\n",\
        __FILE__, __LINE__, clean_errno(), ##__VA_ARGS__)

#define log_info(M, ...) fprintf(stderr, "[INFO] (%s:%d) " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)

#define check(A, M, ...) if(!(A)) {\
    log_err(M, ##__VA_ARGS__); errno=0; goto error; }

#define sentinel(M, ...)  { log_err(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#define check_mem(A) check((A), "Out of memory.")

#define check_debug(A, M, ...) if(!(A)) { debug(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#endif

```

.\ex50b\statserve\src\net.c

```c
#include <stdlib.h>
#include <sys/select.h>
#include <stdio.h>
#include <lcthw/ringbuffer.h>
#include <lcthw/dbg.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <sys/uio.h>
#include <arpa/inet.h>
#include <netdb.h>
#include <unistd.h>
#include <fcntl.h>
#include "net.h"

struct tagbstring NL = bsStatic("\n");
struct tagbstring CRLF = bsStatic("\r\n");

int nonblock(int fd)
{
    int flags = fcntl(fd, F_GETFL, 0);
    check(flags >= 0, "Invalid flags on nonblock.");

    int rc = fcntl(fd, F_SETFL, flags | O_NONBLOCK);
    check(rc == 0, "Can't set nonblocking.");

    return 0;
error:
    return -1;
}

int client_connect(char *host, char *port)
{
    int rc = 0;
    struct addrinfo *addr = NULL;

    rc = getaddrinfo(host, port, NULL, &addr);
    check(rc == 0, "Failed to lookup %s:%s", host, port);

    int sock = socket(AF_INET, SOCK_STREAM, 0);
    check(sock >= 0, "Cannot create a socket.");

    rc = connect(sock, addr->ai_addr, addr->ai_addrlen);
    check(rc == 0, "Connect failed.");

    rc = nonblock(sock);
    check(rc == 0, "Can't set nonblocking.");

    freeaddrinfo(addr);
    return sock;

error:
    freeaddrinfo(addr);
    return -1;
}

int read_some(RingBuffer * buffer, int fd, int is_socket)
{
    int rc = 0;

    if (RingBuffer_available_data(buffer) == 0) {
        buffer->start = buffer->end = 0;
    }

    if (is_socket) {
        rc = recv(fd, RingBuffer_starts_at(buffer),
                RingBuffer_available_space(buffer), 0);
    } else {
        rc = read(fd, RingBuffer_starts_at(buffer),
                RingBuffer_available_space(buffer));
    }

    check(rc >= 0, "Failed to read from fd: %d", fd);

    RingBuffer_commit_write(buffer, rc);

    return rc;

error:
    return -1;
}

int write_some(RingBuffer * buffer, int fd, int is_socket)
{
    int rc = 0;
    bstring data = RingBuffer_get_all(buffer);

    check(data != NULL, "Failed to get from the buffer.");
    check(bfindreplace(data, &NL, &CRLF, 0) == BSTR_OK,
            "Failed to replace NL.");

    if (is_socket) {
        rc = send(fd, bdata(data), blength(data), 0);
    } else {
        rc = write(fd, bdata(data), blength(data));
    }

    check(rc == blength(data), "Failed to write everything to fd: %d.",
            fd);
    bdestroy(data);

    return rc;

error:
    return -1;
}

int attempt_listen(struct addrinfo *info)
{
    int sockfd = -1; // default fail
    int rc = -1;
    int yes = 1;

    check(info != NULL, "Invalid addrinfo.");

    // create a socket with the addrinfo
    sockfd = socket(info->ai_family, info->ai_socktype,
            info->ai_protocol);
    check_debug(sockfd != -1, "Failed to bind to address. Trying more.");

    // set the SO_REUSEADDR option on the socket
    rc = setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(int));
    check_debug(rc == 0, "Failed to set SO_REUSADDR.");

    // attempt to bind to it
    rc = bind(sockfd, info->ai_addr, info->ai_addrlen);
    check_debug(rc == 0, "Failed to find socket.");

    // finally listen with a backlog
    rc = listen(sockfd, BACKLOG);
    check_debug(rc == 0, "Failed to listen to socket.");

    return sockfd;

error:
    return -1;
}

int server_listen(const char *host, const char *port)
{
    int rc = 0;
    int sockfd = -1; // default fail value
    struct addrinfo *info = NULL;
    struct addrinfo *next_p = NULL;
    struct addrinfo addr = {
        .ai_family = AF_UNSPEC,
        .ai_socktype = SOCK_STREAM,
        .ai_flags = AI_PASSIVE
    };

    check(host != NULL, "Invalid host.");
    check(port != NULL, "Invalid port.");

    // get the address info for host and port
    rc = getaddrinfo(NULL, port, &addr, &info);
    check(rc == 0, "Failed to get address info for connect.");

    // cycle through the available list to find one
    for(next_p = info; next_p != NULL; next_p = next_p->ai_next)
    {
        // attempt to listen to each one
        sockfd = attempt_listen(next_p);
        if(sockfd != -1) break;
    }

    // either we found one and were able to listen or nothing.
    check(sockfd != -1, "All possible addresses failed.");

error: //fallthrough
    if(info) freeaddrinfo(info);
    // this gets set by the above to either -1 or valid
    return sockfd;
}

bstring read_line(RingBuffer *input, const char line_ending)
{
    int i = 0;
    bstring result = NULL;

    // not super efficient
    // read a character at a time from the ring buffer
    for(i = 0; i < RingBuffer_available_data(input); i++) {
        // if the buffer has line ending
        if(input->buffer[i] == line_ending) {
            // get that much fromt he ring buffer
            result = RingBuffer_gets(input, i);
            check(result, "Failed to get line from RingBuffer");
            // make sure that we got the right amount
            check(RingBuffer_available_data(input) >= 1, 
                    "Not enough data in the RingBuffer after reading line.");
            // and commit it
            RingBuffer_commit_read(input, 1);
            break;
        }
    }

    // notice this will fail in the cases where we get a set of data
    // on the wire that does not have a line ending yet
    return result;
error:
    return NULL;
}

void send_reply(RingBuffer *send_rb, bstring reply)
{
    RingBuffer_puts(send_rb, reply);
}

```

.\ex50b\statserve\src\net.h

```c
#ifndef _net_h
#define _net_h

#include <lcthw/ringbuffer.h>

#define BACKLOG 10

int nonblock(int fd);
int client_connect(char *host, char *port);
int read_some(RingBuffer * buffer, int fd, int is_socket);
int write_some(RingBuffer * buffer, int fd, int is_socket);
int server_listen(const char *host, const char *port);
bstring read_line(RingBuffer *input, const char line_ending);
void send_reply(RingBuffer *send_rb, bstring reply);

#endif

```

.\ex50b\statserve\src\statserve.c

```c
#include <stdio.h>
#include <ctype.h>
#include <lcthw/dbg.h>
#include <lcthw/hashmap.h>
#include <unistd.h>
#include <stdlib.h>
#include <signal.h>
#include <sys/wait.h>
#include "net.h"
#include <netdb.h>
#include "statserve.h"

struct tagbstring LINE_SPLIT = bsStatic(" ");
struct tagbstring CREATE = bsStatic("create");
struct tagbstring STDDEV = bsStatic("stddev");
struct tagbstring MEAN = bsStatic("mean");
struct tagbstring SAMPLE = bsStatic("sample");
struct tagbstring DUMP = bsStatic("dump");
struct tagbstring DELETE = bsStatic("delete");
struct tagbstring OK = bsStatic("OK\n");
struct tagbstring ERR = bsStatic("ERR\n");
struct tagbstring DNE = bsStatic("DNE\n");
struct tagbstring EXISTS = bsStatic("EXISTS\n");
struct tagbstring SLASH = bsStatic("/");
const char LINE_ENDING = '\n';

const int RB_SIZE = 1024 * 10;

Hashmap *DATA = NULL;

void handle_sigchild(int sig) {
    sig = 0; // ignore it
    while(waitpid(-1, NULL, WNOHANG) > 0) {
    }
}

int handle_create(Command *cmd, RingBuffer *send_rb, bstring path)
{
    int rc = 0;
    int is_root = biseq(path, cmd->name);
    log_info("create: %s %s %s", bdata(cmd->name), bdata(path), bdata(cmd->number));

    Record *info = Hashmap_get(DATA, path);

    if(info != NULL && is_root) {
        // report if root exists, just skip children
        send_reply(send_rb, &EXISTS);
    } else if(info != NULL) {
        debug("Child %s exists, skipping it.", bdata(path));
        return 0;
    } else {
        // new child so make it
        debug("create: %s %s", bdata(path), bdata(cmd->number));

        Record *info = calloc(sizeof(Record), 1);
        check_mem(info);

        // set its stat element
        info->stat = Stats_create();
        check_mem(info->stat);

        // set its name element
        info->name = bstrcpy(path);
        check_mem(info->name);

        // do a first sample
        Stats_sample(info->stat, atof(bdata(cmd->number)));

        // add it to the hashmap
        rc = Hashmap_set(DATA, info->name, info);
        check(rc == 0, "Failed to add data to map.");

        // only send the for the root part
        if(is_root) {
            send_reply(send_rb, &OK);
        }
    }

    return 0;
error:
    return -1;
}

int handle_sample(Command *cmd, RingBuffer *send_rb, bstring path)
{
    // get the info from the hashmap
    Record *info = Hashmap_get(DATA, path);
    int is_root = biseq(path, cmd->name);
    log_info("sample %s %s %s", bdata(cmd->name), bdata(path), bdata(cmd->number));
    bstring child_path = NULL;

    if(info == NULL) {
        // if it doesn't exist then DNE
        send_reply(send_rb, &DNE);
        return 0;
    } else {
        if(is_root) {
            // just sample the root like normal
            Stats_sample(info->stat, atof(bdata(cmd->number)));
        } else {
            // need to do some hackery to get the child path
            // for rolling up mean-of-means on it

            // increase the qty on path up one
            cmd->path->qty++;
            // get the "child path" (previous path?)
            child_path = bjoin(cmd->path, &SLASH);
            // get that info from the DATA
            Record *child_info = Hashmap_get(DATA, child_path);
            bdestroy(child_path);

            // if it exists then sample on it
            if(child_info) {
                // info is /logins, child_info is /logins/zed 
                // we want /logins/zed's mean to be a new sample on /logins
                Stats_sample(info->stat, Stats_mean(child_info->stat));
            }
            // drop the path back to where it was
            cmd->path->qty--;
        }

    }

    // do the reply for the mean last
    bstring reply = bformat("%f\n", Stats_mean(info->stat));
    send_reply(send_rb, reply);
    bdestroy(reply);

    return 0;
}

int handle_delete(Command *cmd, RingBuffer *send_rb, bstring path)
{
    log_info("delete: %s %s", bdata(cmd->name), bdata(path));
    Record *info = Hashmap_get(DATA, path);
    int is_root = biseq(path, cmd->name);

    // BUG: should just decide that this isn't scanned 
    // but run once, for now just only run on root
    if(info == NULL) {
        send_reply(send_rb, &DNE);
    } else if(is_root) {
        Hashmap_delete(DATA, path);

        free(info->stat);
        bdestroy(info->name);
        free(info);

        send_reply(send_rb, &OK);
    }

    return 0;
}

int handle_mean(Command *cmd, RingBuffer *send_rb, bstring path)
{
    log_info("mean: %s %s %s", bdata(cmd->name), bdata(path), bdata(path));
    Record *info = Hashmap_get(DATA, path);

    if(info == NULL) {
        send_reply(send_rb, &DNE);
    } else {
        bstring reply = bformat("%f\n", Stats_mean(info->stat));
        send_reply(send_rb, reply);
        bdestroy(reply);
    }

    return 0;
}

int handle_stddev(Command *cmd, RingBuffer *send_rb, bstring path)
{
    log_info("stddev: %s %s %s", bdata(cmd->name), bdata(path), bdata(path));
    Record *info = Hashmap_get(DATA, path);

    if(info == NULL) {
        send_reply(send_rb, &DNE);
    } else {
        bstring reply = bformat("%f\n", Stats_stddev(info->stat));
        send_reply(send_rb, reply);
        bdestroy(reply);
    }

    return 0;
}

int handle_dump(Command *cmd, RingBuffer *send_rb, bstring path)
{
    log_info("dump: %s, %s, %s", bdata(cmd->name), bdata(path), bdata(path));
    Record *info = Hashmap_get(DATA, path);

    if(info == NULL) {
        send_reply(send_rb, &DNE);
    } else {
        bstring reply = bformat("%f %f %f %f %ld %f %f\n",
                Stats_mean(info->stat),
                Stats_stddev(info->stat),
                info->stat->sum,
                info->stat->sumsq,
                info->stat->n,
                info->stat->min,
                info->stat->max);

        send_reply(send_rb, reply);
        bdestroy(reply);
    }

    return 0;
}

int parse_command(struct bstrList *splits, Command *cmd)
{
    // get the command
    cmd->command = splits->entry[0];

    if(biseq(cmd->command, &CREATE)) {
        check(splits->qty == 3, "Failed to parse create: %d", splits->qty);
        cmd->name = splits->entry[1];
        cmd->number = splits->entry[2];
        cmd->handler = handle_create;
    } else if(biseq(cmd->command, &MEAN)) {
        check(splits->qty == 2, "Failed to parse mean: %d", splits->qty);
        cmd->name = splits->entry[1];
        cmd->handler = handle_mean;
    } else if(biseq(cmd->command, &SAMPLE)) {
        check(splits->qty == 3, "Failed to parse sample: %d", splits->qty);
        cmd->name = splits->entry[1];
        cmd->number = splits->entry[2];
        cmd->handler = handle_sample;
    } else if(biseq(cmd->command, &DUMP)) {
        check(splits->qty == 2, "Failed to parse dump: %d", splits->qty);
        cmd->name = splits->entry[1];
        cmd->handler = handle_dump;
    } else if(biseq(cmd->command, &DELETE)) {
        check(splits->qty == 2, "Failed to parse delete: %d", splits->qty);
        cmd->name = splits->entry[1];
        cmd->handler = handle_delete;
    } else if(biseq(cmd->command, &STDDEV)) {
        check(splits->qty == 2, "Failed to parse stddev: %d", splits->qty);
        cmd->name = splits->entry[1];
        cmd->handler = handle_stddev;
    } else {
        sentinel("Failed to parse the command.");
    }

    return 0;
error:
    return -1;
}

int scan_paths(Command *cmd, RingBuffer *send_rb)
{
    check(cmd->path != NULL, "Path was not set in command.");

    int rc = 0;
    // save the original path length
    size_t qty = cmd->path->qty;

    // starting at the longest path, shorten it and call
    // for each one:
    for(; cmd->path->qty > 1; cmd->path->qty--) {
        // remake the path with / again
        bstring path = bjoin(cmd->path, &SLASH);
        // call the handler with the path
        rc = cmd->handler(cmd, send_rb, path);
        // if the handler returns != 0 then abort and return that
        bdestroy(path);

        if(rc != 0) break;
    }

    // restore path length
    cmd->path->qty = qty;
    return rc;
error:
    return -1;
}

struct bstrList *parse_name(bstring name)
{
    return bsplits(name, &SLASH);
}

int parse_line(bstring data, RingBuffer *send_rb)
{
    int rc = -1;
    Command cmd = {.command = NULL};

    // split data on line boundaries
    struct bstrList *splits = bsplits(data, &LINE_SPLIT);
    check(splits != NULL, "Bad data.");

    // parse it into a command
    rc = parse_command(splits, &cmd);
    check(rc == 0, "Failed to parse command.");

    // parse the name into the path we need for scan_paths
    cmd.path = parse_name(cmd.name);
    check(cmd.path != NULL, "Invalid path.");

    // scan the path and call the handlers
    rc = scan_paths(&cmd, send_rb);
    check(rc == 0, "Failure running command against path: %s", bdata(cmd.name));

    bstrListDestroy(cmd.path);
    bstrListDestroy(splits);

    return 0;

error: // fallthrough
    if(cmd.path) bstrListDestroy(cmd.path);
    if(splits) bstrListDestroy(splits);
    return -1;
}

void client_handler(int client_fd)
{
    int rc = 0;
    RingBuffer *recv_rb = RingBuffer_create(RB_SIZE);
    RingBuffer *send_rb = RingBuffer_create(RB_SIZE);

    check_mem(recv_rb);
    check_mem(send_rb);

    // keep reading into the recv buffer and sending on send
    while(read_some(recv_rb, client_fd, 1) != -1) {
        // read a line from the recv_rb
        bstring data = read_line(recv_rb, LINE_ENDING);
        check(data != NULL, "Client closed.");

        // parse it, close on any protocol errors
        rc = parse_line(data, send_rb);
        bdestroy(data); // cleanup here
        check(rc == 0, "Failed to parse user. Closing.");

        // and as long as there's something to send, send it
        if(RingBuffer_available_data(send_rb)) {
            write_some(send_rb, client_fd, 1);
        }
    }

    // close the socket
    rc = close(client_fd);
    check(rc != -1, "Failed to close the socket.");

error: // fallthrough
    if(recv_rb) RingBuffer_destroy(recv_rb);
    if(send_rb) RingBuffer_destroy(send_rb);
    exit(0); // just exit the child process
}

int setup_data_store()
{
    // a more advanced design simply wouldn't use this
    DATA = Hashmap_create(NULL, NULL);
    check_mem(DATA);

    return 0;
error:
    return -1;
}

int echo_server(const char *host, const char *port)
{
    int rc = 0;
    struct sockaddr_in client_addr;
    socklen_t sin_size = sizeof(client_addr);
    int server_socket = 0;
    int client_fd = 0;

    rc = setup_data_store();
    check(rc == 0, "Failed to setup the data store.");

    struct sigaction sa = {
        .sa_handler = handle_sigchild,
        .sa_flags = SA_RESTART | SA_NOCLDSTOP
    };

    check(host != NULL, "Invalid host.");
    check(port != NULL, "Invalid port.");

    // create a sigaction that handles SIGCHLD
    sigemptyset(&sa.sa_mask);
    rc = sigaction(SIGCHLD, &sa, 0);
    check(rc != -1, "Failed to setup signal handler for child processes.");

    // listen on the given port and host
    server_socket = server_listen(host, port);
    check(server_socket >= 0, "bind to %s:%s failed.", host, port);

    while(1) {
        // accept the connection
        client_fd = accept(server_socket, (struct sockaddr *)&client_addr, &sin_size); 
        check(client_fd >= 0, "Failed to accept connection.");

        debug("Client connected.");

        rc = fork();
        if(rc == 0) {
            // child process
            close(server_socket); // don't need this
            // handle the client
            client_handler(client_fd);
        } else {
            // server process
            close(client_fd); // don't need this
        }
    }

error:  // fallthrough
    return -1;
}

```

.\ex50b\statserve\src\statserve.h

```c
#ifndef _statserve_h
#define _statserve_h

#include <lcthw/bstrlib.h>
#include <lcthw/ringbuffer.h>
#include <lcthw/stats.h>

struct Command;

typedef int (*handler_cb)(struct Command *cmd, RingBuffer *send_rb, bstring path);

typedef struct Command {
    bstring command;
    bstring name;
    struct bstrList *path;
    bstring number;
    handler_cb handler;
} Command;

typedef struct Record {
    bstring name;
    Stats *stat;
} Record;

struct tagbstring OK;

int setup_data_store();

struct bstrList *parse_name(bstring name);

int scan_paths(Command *cmd, RingBuffer *send_rb);

int parse_line(bstring data, RingBuffer *send_rb);

int echo_server(const char *host, const char *port);

#endif

```

.\ex50b\statserve\tests\statserve_tests.c

```c
#include "minunit.h"
#include <dlfcn.h>
#include "statserve.h"
#include <lcthw/bstrlib.h>
#include <lcthw/ringbuffer.h>
#include <assert.h>

typedef struct LineTest {
    char *line;
    bstring result;
    char *description;
} LineTest;

int attempt_line(LineTest test)
{
    int rc = -1;
    bstring result = NULL;

    bstring line = bfromcstr(test.line);
    RingBuffer *send_rb = RingBuffer_create(1024);

    rc = parse_line(line, send_rb);
    check(rc == 0, "Failed to parse line.");

    result = RingBuffer_get_all(send_rb);
    check(result != NULL, "Ring buffer empty.");
    check(biseq(result, test.result), "Got the wrong output: %s expected %s",
            bdata(result), bdata(test.result));

    bdestroy(line);
    RingBuffer_destroy(send_rb);
    return 1; // using 1 for tests
error:

    log_err("Failed to process test %s: got %s", test.line, bdata(result));
    if(line) bdestroy(line);
    if(send_rb) RingBuffer_destroy(send_rb);
    return 0;
}

int run_test_lines(LineTest *tests, int count)
{
    int i = 0;

    for(i = 0; i < count; i++) {
        check(attempt_line(tests[i]), "Failed to run %s", tests[i].description);
    }

    return 1;
error:
    return 0;
}

int fake_command(Command *cmd, RingBuffer *send_rb, bstring path)
{
    check(cmd != NULL, "Bad cmd.");
    check(cmd->path != NULL, "Bad path.");
    check(send_rb != NULL, "Bad send_rb.");
    check(path != NULL, "Bad path given.");

    return 0;
error:
    return -1;
}

char *test_path_parsing()
{
    struct bstrList *result = NULL;
    struct tagbstring slash = bsStatic("/");
    struct tagbstring logins_zed = bsStatic("/logins/zed");
    struct tagbstring command_name = bsStatic("dump");
    RingBuffer *send_rb = RingBuffer_create(1024);
    struct bstrList *path = bsplits(&logins_zed, &slash);
    int rc = 0;

    Command fake = {
        .command = &command_name,
        .name = &logins_zed,
        .number = NULL,
        .handler = fake_command,
        .path = path
    };

    result = parse_name(&logins_zed);
    mu_assert(result != NULL, "Failed to parse /logins/zed");

    rc = scan_paths(&fake, send_rb); 
    mu_assert(rc != -1, "scan_paths failed.");

    return NULL;
}

char *test_create()
{
    LineTest tests[] = {
        {.line = "create /zed 100", .result = &OK, .description = "create zed failed"},
        {.line = "create /joe 100", .result = &OK, .description = "create joe failed"},

    };

    mu_assert(run_test_lines(tests, 2), "Failed to run create tests.");

    return NULL;
}

char *test_sample()
{
    struct tagbstring sample1 = bsStatic("100.000000\n");

    LineTest tests[] = {
        {.line = "sample /zed 100", .result = &sample1, .description = "sample zed failed."}
    };

    mu_assert(run_test_lines(tests, 1), "Failed to run sample tests.");

    return NULL;
}

char *all_tests()
{
    mu_suite_start();

    int rc = setup_data_store();
    mu_assert(rc == 0, "Failed to setup the data store.");

    mu_run_test(test_create);
    mu_run_test(test_sample);
    mu_run_test(test_path_parsing);

    return NULL;
}

RUN_TESTS(all_tests);

```

Final Review

### Exercise 51a Storing the Statistics

Project Description

The Plan

Learn to store the statistics to the hard disk.

There are meany issues with this.

The Purpose

To teach you about various problems related to securely storing files.

The Requirements

For this exercise, you'll add two commands for storing to and loading statistics
from a hard drive:

store
    If there's a URL, store it to a hard drive.

load
    If there are two URLs, load the statistic from the hard drive based on the first URL, and then put it into the second URL that's in memory.

The Requirements

1. If URLs have ``/`` characters in them, then that conflicts with the filesystem's use of slashes.  How will you solve this?
2. If URLs have ``/`` characters in them, then someone can use your server to overwrite files on a hard drive by giving paths to them.  How will you solve this?
3. If you choose to use deeply nested directories, then traversing directories to find files will be very slow.  What will you do here?

The Requirements

4. If you choose to use one directory and hash URLs (oops, I gave a hint), then directories with too many files in them are slow.  How will you solve this?
5. What happens when someone loads a statistic from a hard drive into a URL that already exists?
6. How will someone running ``statserve`` know where the storage should be?

The Clues

There are no clues.  You can do this.

### Exercise 51b Storing the Statistics

Solution

The Plan

Show you how I solved the problem of storing the statistics to disk.

Security Requirements

* Use *realpath* to make sure that paths are in one place.
* Use _BAD_ encryption to mangle the stored names.
* No other security beyond that. Just a demo of the path issue.

XTEA Encryption

* For an extra challenge I decided to "hash" names with XTEA.
* https://en.wikipedia.org/wiki/XTEA for the code.
* Normally I wouldn't do this, but wanted to show you XTEA.
* Because XTEA if cool and fun, although broken.
* DON'T USE XTEA FOR ENCRYPTION.

Improvements

* Let commands set cmd->path = NULL to indicate non-recursive.
* Change *echo_server* to *run_server* finally.
* Allow a 3rd storage path argument on the command line.
* Allow an additional argument to Command.

Weirdness

* Forking means I can't share data between clients without storage.
* Storing doesn't happen automatically, only explicitly.
* Loading acts as a copy command.
* XTEA isn't the best algorithm at all.  Just for fun.

How I Did It

1. Create the LOAD and STORE handlers.
2. Add Command.arg and set those in *parse\_command*.
3. Move path parsing up to allow non-recursive handling with path = NULL.
4. Write a *sanitize\_location* and *encrypt\_armor\_name* function, test them.
5. Write *handle\_store* first to allow testing *handle\_load*.
6. Use *open* (man 2 open) with O_EXLOCK to get exclusive locks on files.
7. Using *close* (man 2 close) should release EXLOCK, but not clear on this.

The Code

.\ex51b\statserve

.\ex51b\statserve\bin\statserve.c

```c
#include <stdio.h>
#include <lcthw/dbg.h>
#include "statserve.h"
#include "net.h"

int main(int argc, char *argv[])
{
    check(argc == 4, "USAGE: statserve host port store_path");

    const char *host = argv[1];
    const char *port = argv[2];
    const char *store_path = argv[3];

    check(run_server(host, port, store_path), "Failed to run the echo server.");

    return 0;

error:

    return 1;
}

```

.\ex51b\statserve\src\dbg.h

```c
#ifndef __dbg_h__
#define __dbg_h__

#include <stdio.h>
#include <errno.h>
#include <string.h>

#ifdef NDEBUG
#define debug(M, ...)
#else
#define debug(M, ...) fprintf(stderr, "DEBUG %s:%d: " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)
#endif

#define clean_errno() (errno == 0 ? "None" : strerror(errno))

#define log_err(M, ...) fprintf(stderr,\
        "[ERROR] (%s:%d: errno: %s) " M "\n", __FILE__, __LINE__,\
        clean_errno(), ##__VA_ARGS__)

#define log_warn(M, ...) fprintf(stderr,\
        "[WARN] (%s:%d: errno: %s) " M "\n",\
        __FILE__, __LINE__, clean_errno(), ##__VA_ARGS__)

#define log_info(M, ...) fprintf(stderr, "[INFO] (%s:%d) " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)

#define check(A, M, ...) if(!(A)) {\
    log_err(M, ##__VA_ARGS__); errno=0; goto error; }

#define sentinel(M, ...)  { log_err(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#define check_mem(A) check((A), "Out of memory.")

#define check_debug(A, M, ...) if(!(A)) { debug(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#endif

```

.\ex51b\statserve\src\net.c

```c
#include <stdlib.h>
#include <sys/select.h>
#include <stdio.h>
#include <lcthw/ringbuffer.h>
#include <lcthw/dbg.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <sys/uio.h>
#include <arpa/inet.h>
#include <netdb.h>
#include <unistd.h>
#include <fcntl.h>
#include "net.h"

struct tagbstring NL = bsStatic("\n");
struct tagbstring CRLF = bsStatic("\r\n");

int nonblock(int fd)
{
    int flags = fcntl(fd, F_GETFL, 0);
    check(flags >= 0, "Invalid flags on nonblock.");

    int rc = fcntl(fd, F_SETFL, flags | O_NONBLOCK);
    check(rc == 0, "Can't set nonblocking.");

    return 0;
error:
    return -1;
}

int client_connect(char *host, char *port)
{
    int rc = 0;
    struct addrinfo *addr = NULL;

    rc = getaddrinfo(host, port, NULL, &addr);
    check(rc == 0, "Failed to lookup %s:%s", host, port);

    int sock = socket(AF_INET, SOCK_STREAM, 0);
    check(sock >= 0, "Cannot create a socket.");

    rc = connect(sock, addr->ai_addr, addr->ai_addrlen);
    check(rc == 0, "Connect failed.");

    rc = nonblock(sock);
    check(rc == 0, "Can't set nonblocking.");

    freeaddrinfo(addr);
    return sock;

error:
    freeaddrinfo(addr);
    return -1;
}

int read_some(RingBuffer * buffer, int fd, int is_socket)
{
    int rc = 0;

    if (RingBuffer_available_data(buffer) == 0) {
        buffer->start = buffer->end = 0;
    }

    if (is_socket) {
        rc = recv(fd, RingBuffer_starts_at(buffer),
                RingBuffer_available_space(buffer), 0);
    } else {
        rc = read(fd, RingBuffer_starts_at(buffer),
                RingBuffer_available_space(buffer));
    }

    check(rc >= 0, "Failed to read from fd: %d", fd);

    RingBuffer_commit_write(buffer, rc);

    return rc;

error:
    return -1;
}

int write_some(RingBuffer * buffer, int fd, int is_socket)
{
    int rc = 0;
    bstring data = RingBuffer_get_all(buffer);

    check(data != NULL, "Failed to get from the buffer.");
    check(bfindreplace(data, &NL, &CRLF, 0) == BSTR_OK,
            "Failed to replace NL.");

    if (is_socket) {
        rc = send(fd, bdata(data), blength(data), 0);
    } else {
        rc = write(fd, bdata(data), blength(data));
    }

    check(rc == blength(data), "Failed to write everything to fd: %d.",
            fd);
    bdestroy(data);

    return rc;

error:
    return -1;
}

int attempt_listen(struct addrinfo *info)
{
    int sockfd = -1; // default fail
    int rc = -1;
    int yes = 1;

    check(info != NULL, "Invalid addrinfo.");

    // create a socket with the addrinfo
    sockfd = socket(info->ai_family, info->ai_socktype,
            info->ai_protocol);
    check_debug(sockfd != -1, "Failed to bind to address. Trying more.");

    // set the SO_REUSEADDR option on the socket
    rc = setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(int));
    check_debug(rc == 0, "Failed to set SO_REUSADDR.");

    // attempt to bind to it
    rc = bind(sockfd, info->ai_addr, info->ai_addrlen);
    check_debug(rc == 0, "Failed to find socket.");

    // finally listen with a backlog
    rc = listen(sockfd, BACKLOG);
    check_debug(rc == 0, "Failed to listen to socket.");

    return sockfd;

error:
    return -1;
}

int server_listen(const char *host, const char *port)
{
    int rc = 0;
    int sockfd = -1; // default fail value
    struct addrinfo *info = NULL;
    struct addrinfo *next_p = NULL;
    struct addrinfo addr = {
        .ai_family = AF_UNSPEC,
        .ai_socktype = SOCK_STREAM,
        .ai_flags = AI_PASSIVE
    };

    check(host != NULL, "Invalid host.");
    check(port != NULL, "Invalid port.");

    // get the address info for host and port
    rc = getaddrinfo(NULL, port, &addr, &info);
    check(rc == 0, "Failed to get address info for connect.");

    // cycle through the available list to find one
    for(next_p = info; next_p != NULL; next_p = next_p->ai_next)
    {
        // attempt to listen to each one
        sockfd = attempt_listen(next_p);
        if(sockfd != -1) break;
    }

    // either we found one and were able to listen or nothing.
    check(sockfd != -1, "All possible addresses failed.");

error: //fallthrough
    if(info) freeaddrinfo(info);
    // this gets set by the above to either -1 or valid
    return sockfd;
}

bstring read_line(RingBuffer *input, const char line_ending)
{
    int i = 0;
    bstring result = NULL;

    // not super efficient
    // read a character at a time from the ring buffer
    for(i = 0; i < RingBuffer_available_data(input); i++) {
        // if the buffer has line ending
        if(input->buffer[i] == line_ending) {
            // get that much fromt he ring buffer
            result = RingBuffer_gets(input, i);
            check(result, "Failed to get line from RingBuffer");
            // make sure that we got the right amount
            check(RingBuffer_available_data(input) >= 1, 
                    "Not enough data in the RingBuffer after reading line.");
            // and commit it
            RingBuffer_commit_read(input, 1);
            break;
        }
    }

    // notice this will fail in the cases where we get a set of data
    // on the wire that does not have a line ending yet
    return result;
error:
    return NULL;
}

void send_reply(RingBuffer *send_rb, bstring reply)
{
    RingBuffer_puts(send_rb, reply);
}

```

.\ex51b\statserve\src\net.h

```c
#ifndef _net_h
#define _net_h

#include <lcthw/ringbuffer.h>

#define BACKLOG 10

int nonblock(int fd);
int client_connect(char *host, char *port);
int read_some(RingBuffer * buffer, int fd, int is_socket);
int write_some(RingBuffer * buffer, int fd, int is_socket);
int server_listen(const char *host, const char *port);
bstring read_line(RingBuffer *input, const char line_ending);
void send_reply(RingBuffer *send_rb, bstring reply);

#endif

```

.\ex51b\statserve\src\statserve.c

```c
#include <stdio.h>
#include <ctype.h>
#include <lcthw/dbg.h>
#include <lcthw/hashmap.h>
#include <unistd.h>
#include <stdlib.h>
#include <signal.h>
#include <sys/wait.h>
#include "net.h"
#include <netdb.h>
#include <fcntl.h>
#include "statserve.h"

struct tagbstring LINE_SPLIT = bsStatic(" ");
struct tagbstring CREATE = bsStatic("create");
struct tagbstring STDDEV = bsStatic("stddev");
struct tagbstring MEAN = bsStatic("mean");
struct tagbstring SAMPLE = bsStatic("sample");
struct tagbstring DUMP = bsStatic("dump");
struct tagbstring DELETE = bsStatic("delete");
struct tagbstring STORE = bsStatic("store");
struct tagbstring LOAD = bsStatic("load");
struct tagbstring OK = bsStatic("OK\n");
struct tagbstring ERR = bsStatic("ERR\n");
struct tagbstring DNE = bsStatic("DNE\n");
struct tagbstring EXISTS = bsStatic("EXISTS\n");
struct tagbstring SLASH = bsStatic("/");
const char LINE_ENDING = '\n';

const int RB_SIZE = 1024 * 10;

Hashmap *DATA = NULL;
bstring STORE_PATH = NULL;

void handle_sigchild(int sig) {
    sig = 0; // ignore it
    while(waitpid(-1, NULL, WNOHANG) > 0) {
    }
}

// BUG: this is stupid, use md5
void encipher(unsigned int num_rounds, uint32_t v[2], uint32_t const key[4]) {
    unsigned int i;
    uint32_t v0=v[0], v1=v[1], sum=0, delta=0x9E3779B9;
    for (i=0; i < num_rounds; i++) {
        v0 += (((v1 << 4) ^ (v1 >> 5)) + v1) ^ (sum + key[sum & 3]);
        sum += delta;
        v1 += (((v0 << 4) ^ (v0 >> 5)) + v0) ^ (sum + key[(sum>>11) & 3]);
    }
    v[0]=v0; v[1]=v1;
}

/// TOTALLY RANDOM! LOL  BUG: not secure
const uint32_t STORE_KEY[4] = {18748274, 228374, 193034845, 85726348};
struct tagbstring FAUX64 = bsStatic("ABCDEFGHIJKLMNOPQRST<KEY>");

// BUG: this all dies
bstring encrypt_armor_name(bstring name)
{
    // copy the name to encrypt
    bstring encname = bstrcpy(name);
    size_t i = 0;
    // point the encrypt pointer at it
    // BUG: this cast is weird, why?
    uint32_t *v = (uint32_t *)bdata(encname);

    // extend the encname so that it can hold everything
    // BUG: use a correct padding algorithm
    while(blength(encname) % (sizeof(uint32_t) * 2) > 0) {
        bconchar(encname, ' ');
    }

    // run encipher on this
    // BUG: get rid of encipher
    for(i = 0; i < (size_t)blength(encname) / (sizeof(uint32_t) * 2); i+=2) {
        encipher(1, v+i, STORE_KEY);
    }

    // do a lame "base 64" kind of thing on it
    // BUG: this is NOT the best way, it's a quick hack to get it working
    // replace with real BASE64 later
    for(i = 0; i < (size_t)blength(encname); i++) {
        int at = encname->data[i] % blength(&FAUX64);
        encname->data[i] = FAUX64.data[at];
    }

    // that's our final hack encrypted name
    return encname;
}

bstring sanitize_location(bstring base, bstring path)
{
    bstring attempt = NULL;
    bstring encpath = NULL;

    // encrypt armore the name
    // BUG: ditch encryption, it was dumb
    encpath = encrypt_armor_name(path);
    check(encpath != NULL, "Failed to encrypt path name: %s", bdata(path));

    // combine it with the base, this means that we've armored the 
    // path so we can just append it
    attempt = bformat("%s/%s", bdata(base), bdata(encpath));
    bdestroy(encpath);
    return attempt;

error:
    if(encpath) bdestroy(encpath);
    if(attempt) bdestroy(attempt);
    return NULL;
}

int handle_create(Command *cmd, RingBuffer *send_rb, bstring path)
{
    int rc = 0;
    int is_root = biseq(path, cmd->name);
    log_info("create: %s %s %s", bdata(cmd->name), bdata(path), bdata(cmd->number));

    Record *info = Hashmap_get(DATA, path);

    if(info != NULL && is_root) {
        // report if root exists, just skip children
        send_reply(send_rb, &EXISTS);
    } else if(info != NULL) {
        debug("Child %s exists, skipping it.", bdata(path));
        return 0;
    } else {
        // new child so make it
        debug("create: %s %s", bdata(path), bdata(cmd->number));

        Record *info = calloc(1, sizeof(Record));
        check_mem(info);

        // set its stat element
        info->stat = Stats_create();
        check_mem(info->stat);

        // set its name element
        info->name = bstrcpy(path);
        check_mem(info->name);

        // do a first sample
        Stats_sample(info->stat, atof(bdata(cmd->number)));

        // add it to the hashmap
        rc = Hashmap_set(DATA, info->name, info);
        check(rc == 0, "Failed to add data to map.");

        // only send the for the root part
        if(is_root) {
            send_reply(send_rb, &OK);
        }
    }

    return 0;
error:
    return -1;
}

int handle_sample(Command *cmd, RingBuffer *send_rb, bstring path)
{
    // get the info from the hashmap
    Record *info = Hashmap_get(DATA, path);
    int is_root = biseq(path, cmd->name);
    log_info("sample %s %s %s", bdata(cmd->name), bdata(path), bdata(cmd->number));
    bstring child_path = NULL;

    if(info == NULL) {
        // if it doesn't exist then DNE
        send_reply(send_rb, &DNE);
        return 0;
    } else {
        if(is_root) {
            // just sample the root like normal
            Stats_sample(info->stat, atof(bdata(cmd->number)));
        } else {
            // need to do some hackery to get the child path
            // for rolling up mean-of-means on it

            // increase the qty on path up one
            cmd->path->qty++;
            // get the "child path" (previous path?)
            child_path = bjoin(cmd->path, &SLASH);
            // get that info from the DATA
            Record *child_info = Hashmap_get(DATA, child_path);
            bdestroy(child_path);

            // if it exists then sample on it
            if(child_info) {
                // info is /logins, child_info is /logins/zed 
                // we want /logins/zed's mean to be a new sample on /logins
                Stats_sample(info->stat, Stats_mean(child_info->stat));
            }
            // drop the path back to where it was
            cmd->path->qty--;
        }

    }

    // do the reply for the mean last
    bstring reply = bformat("%f\n", Stats_mean(info->stat));
    send_reply(send_rb, reply);
    bdestroy(reply);

    return 0;
}

int handle_delete(Command *cmd, RingBuffer *send_rb, bstring path)
{
    log_info("delete: %s", bdata(cmd->name));
    Record *info = Hashmap_get(DATA, cmd->name);
    check(path == NULL && cmd->path == NULL, "Should be a recursive command.");

    // BUG: should just decide that this isn't scanned 
    // but run once, for now just only run on root
    if(info == NULL) {
        send_reply(send_rb, &DNE);
    } else {
        Hashmap_delete(DATA, cmd->name);

        free(info->stat);
        bdestroy(info->name);
        free(info);

        send_reply(send_rb, &OK);
    }

    return 0;
error:
    return -1;
}

int handle_mean(Command *cmd, RingBuffer *send_rb, bstring path)
{
    log_info("mean: %s %s %s", bdata(cmd->name), bdata(path), bdata(path));
    Record *info = Hashmap_get(DATA, path);

    if(info == NULL) {
        send_reply(send_rb, &DNE);
    } else {
        bstring reply = bformat("%f\n", Stats_mean(info->stat));
        send_reply(send_rb, reply);
        bdestroy(reply);
    }

    return 0;
}

int handle_stddev(Command *cmd, RingBuffer *send_rb, bstring path)
{
    log_info("stddev: %s %s %s", bdata(cmd->name), bdata(path), bdata(path));
    Record *info = Hashmap_get(DATA, path);

    if(info == NULL) {
        send_reply(send_rb, &DNE);
    } else {
        bstring reply = bformat("%f\n", Stats_stddev(info->stat));
        send_reply(send_rb, reply);
        bdestroy(reply);
    }

    return 0;
}

int handle_dump(Command *cmd, RingBuffer *send_rb, bstring path)
{
    log_info("dump: %s, %s, %s", bdata(cmd->name), bdata(path), bdata(path));
    Record *info = Hashmap_get(DATA, path);

    if(info == NULL) {
        send_reply(send_rb, &DNE);
    } else {
        bstring reply = bformat("%f %f %f %f %ld %f %f\n",
                Stats_mean(info->stat),
                Stats_stddev(info->stat),
                info->stat->sum,
                info->stat->sumsq,
                info->stat->n,
                info->stat->min,
                info->stat->max);

        send_reply(send_rb, reply);
        bdestroy(reply);
    }

    return 0;
}

int handle_store(Command *cmd, RingBuffer *send_rb, bstring path)
{
    Record *info = Hashmap_get(DATA, cmd->name);
    bstring location = NULL;
    bstring from = cmd->name;
    int rc = 0;
    int fd = -1;

    check(cmd != NULL, "Invalid command.");
    debug("store %s", bdata(cmd->name));
    check(path == NULL && cmd->path == NULL, "Store is non-recursive.");

    if(info == NULL) {
        send_reply(send_rb, &DNE);
    } else {
        // it exists so we sanitize the name
        location = sanitize_location(STORE_PATH, from);
        check(location, "Failed to sanitize the location.");

        // open the file we need with EXLOCK
        fd = open(bdata(location), O_WRONLY | O_CREAT | O_EXLOCK, S_IRWXU);
        check(fd >= 0, "Cannot open file for writing: %s", bdata(location));

        // write the Stats part of info to it
        rc = write(fd, info->stat, sizeof(Stats));
        check(rc == sizeof(Stats), "Failed to write to %s", bdata(location));

        // close, which should release the lock
        close(fd);

        // then send OK
        send_reply(send_rb, &OK);
    }

    return 0;
error: 
    if(fd < 0) close(fd);
    return -1;
}

int handle_load(Command *cmd, RingBuffer *send_rb, bstring path)
{
    bstring to = cmd->arg;
    bstring from = cmd->name;
    bstring location = NULL;
    Record *info = Hashmap_get(DATA, to);
    int fd = -1;

    check(path == NULL && cmd->path == NULL, "Load is non-recursive.");

    if(info != NULL) {
        // don't do it if the target to exists
        send_reply(send_rb, &EXISTS);
    } else {
        location = sanitize_location(STORE_PATH, from);
        check(location, "Failed to sanitize location.");

        // make a new record for the to target
        // TODO: make regular CRUD methods for Record
        info = calloc(1, sizeof(Record));
        check_mem(info);

        info->stat = calloc(1, sizeof(Stats));
        check_mem(info->stat);

        // open the file to read from readonly and locked
        fd = open(bdata(location), O_RDONLY | O_EXLOCK);
        check(fd >= 0, "Error opening file: %s", bdata(location));

        // read into the stats record 
        int rc = read(fd, info->stat, sizeof(Stats));
        check(rc == sizeof(Stats), "Failed to read record at %s", bdata(location));

        // close so we release the lock quick
        close(fd);

        // make a copy of to as the name for the info
        info->name = bstrcpy(to);
        check_mem(info->name);

        // put it in the hashmap
        rc = Hashmap_set(DATA, info->name, info);
        check(rc == 0, "Failed to ass to data map: %s", bdata(info->name));

        // and send the reply
        send_reply(send_rb, &OK);
    }

    return 0;
error:
    if(fd < 0) close(fd);
    return -1;
}

int parse_command(struct bstrList *splits, Command *cmd)
{
    check(splits != NULL, "Invalid split line.");

    // get the command
    cmd->command = splits->entry[0];

    if(biseq(cmd->command, &CREATE)) {
        check(splits->qty == 3, "Failed to parse create: %d", splits->qty);
        cmd->name = splits->entry[1];
        cmd->number = splits->entry[2];
        cmd->handler = handle_create;
        cmd->path = parse_name(cmd->name);
    } else if(biseq(cmd->command, &MEAN)) {
        check(splits->qty == 2, "Failed to parse mean: %d", splits->qty);
        cmd->name = splits->entry[1];
        cmd->handler = handle_mean;
        cmd->path = parse_name(cmd->name);
    } else if(biseq(cmd->command, &SAMPLE)) {
        check(splits->qty == 3, "Failed to parse sample: %d", splits->qty);
        cmd->name = splits->entry[1];
        cmd->number = splits->entry[2];
        cmd->handler = handle_sample;
        cmd->path = parse_name(cmd->name);
    } else if(biseq(cmd->command, &DUMP)) {
        check(splits->qty == 2, "Failed to parse dump: %d", splits->qty);
        cmd->name = splits->entry[1];
        cmd->handler = handle_dump;
        cmd->path = parse_name(cmd->name);
    } else if(biseq(cmd->command, &DELETE)) {
        check(splits->qty == 2, "Failed to parse delete: %d", splits->qty);
        cmd->name = splits->entry[1];
        cmd->handler = handle_delete;
        cmd->path = NULL;
    } else if(biseq(cmd->command, &STDDEV)) {
        check(splits->qty == 2, "Failed to parse stddev: %d", splits->qty);
        cmd->name = splits->entry[1];
        cmd->handler = handle_stddev;
        cmd->path = parse_name(cmd->name);
    } else if(biseq(cmd->command, &STORE)) {
        // store URL
        check(splits->qty == 2, "Failed to parse store: %d", splits->qty);
        cmd->name = splits->entry[1];
        cmd->handler = handle_store;
        cmd->path = NULL;
    } else if(biseq(cmd->command, &LOAD)) {
        // load FROM TO
        check(splits->qty == 3, "Failed to parse load: %d", splits->qty);
        cmd->name = splits->entry[1];
        cmd->arg = splits->entry[2];
        cmd->handler = handle_load;
        cmd->path = NULL;
    } else {
        sentinel("Failed to parse the command.");
    }

    return 0;
error:
    return -1;
}

int scan_paths(Command *cmd, RingBuffer *send_rb)
{
    check(cmd->path != NULL, "Path was not set in command.");

    int rc = 0;
    // save the original path length
    size_t qty = cmd->path->qty;

    // starting at the longest path, shorten it and call
    // for each one:
    for(; cmd->path->qty > 1; cmd->path->qty--) {
        // remake the path with / again
        bstring path = bjoin(cmd->path, &SLASH);
        // call the handler with the path
        rc = cmd->handler(cmd, send_rb, path);
        // if the handler returns != 0 then abort and return that
        bdestroy(path);

        if(rc != 0) break;
    }

    // restore path length
    cmd->path->qty = qty;
    return rc;
error:
    return -1;
}

struct bstrList *parse_name(bstring name)
{
    return bsplits(name, &SLASH);
}

int parse_line(bstring data, RingBuffer *send_rb)
{
    int rc = -1;
    Command cmd = {.command = NULL};

    // split data on line boundaries
    struct bstrList *splits = bsplits(data, &LINE_SPLIT);
    check(splits != NULL, "Bad data.");

    // parse it into a command
    rc = parse_command(splits, &cmd);
    check(rc == 0, "Failed to parse command.");

    // scan the path and call the handlers
    if(cmd.path) { 
        check(cmd.path->qty > 1, "Didn't give a valid URL.");
        rc = scan_paths(&cmd, send_rb);
        check(rc == 0, "Failure running recursive command against path: %s", bdata(cmd.name));
        bstrListDestroy(cmd.path);
    } else {
        rc = cmd.handler(&cmd, send_rb, NULL);
        check(rc == 0, "Failed running command against path: %s", bdata(cmd.name));
    }

    bstrListDestroy(splits);

    return 0;

error: // fallthrough
    if(cmd.path) bstrListDestroy(cmd.path);
    if(splits) bstrListDestroy(splits);
    return -1;
}

void client_handler(int client_fd)
{
    int rc = 0;
    RingBuffer *recv_rb = RingBuffer_create(RB_SIZE);
    RingBuffer *send_rb = RingBuffer_create(RB_SIZE);

    check_mem(recv_rb);
    check_mem(send_rb);

    // keep reading into the recv buffer and sending on send
    while(read_some(recv_rb, client_fd, 1) != -1) {
        // read a line from the recv_rb
        bstring data = read_line(recv_rb, LINE_ENDING);
        check(data != NULL, "Client closed.");

        // parse it, close on any protocol errors
        rc = parse_line(data, send_rb);
        bdestroy(data); // cleanup here
        check(rc == 0, "Failed to parse user. Closing.");

        // and as long as there's something to send, send it
        if(RingBuffer_available_data(send_rb)) {
            write_some(send_rb, client_fd, 1);
        }
    }

    // close the socket
    rc = close(client_fd);
    check(rc != -1, "Failed to close the socket.");

error: // fallthrough
    if(recv_rb) RingBuffer_destroy(recv_rb);
    if(send_rb) RingBuffer_destroy(send_rb);
    exit(0); // just exit the child process
}

int setup_data_store(const char *store_path)
{
    // a more advanced design simply wouldn't use this
    DATA = Hashmap_create(NULL, NULL);
    check_mem(DATA);

    char *path = realpath(store_path, NULL);
    check(path != NULL, "Failed to get the real path for storage: %s", store_path);

    STORE_PATH = bfromcstr(path);
    free(path);

    return 0;
error:
    return -1;
}

int run_server(const char *host, const char *port, const char *store_path)
{
    int rc = 0;
    struct sockaddr_in client_addr;
    socklen_t sin_size = sizeof(client_addr);
    int server_socket = 0;
    int client_fd = 0;

    rc = setup_data_store(store_path);
    check(rc == 0, "Failed to setup the data store.");

    struct sigaction sa = {
        .sa_handler = handle_sigchild,
        .sa_flags = SA_RESTART | SA_NOCLDSTOP
    };

    check(host != NULL, "Invalid host.");
    check(port != NULL, "Invalid port.");

    // create a sigaction that handles SIGCHLD
    sigemptyset(&sa.sa_mask);
    rc = sigaction(SIGCHLD, &sa, 0);
    check(rc != -1, "Failed to setup signal handler for child processes.");

    // listen on the given port and host
    server_socket = server_listen(host, port);
    check(server_socket >= 0, "bind to %s:%s failed.", host, port);

    while(1) {
        // accept the connection
        client_fd = accept(server_socket, (struct sockaddr *)&client_addr, &sin_size); 
        check(client_fd >= 0, "Failed to accept connection.");

        debug("Client connected.");

        rc = fork();
        if(rc == 0) {
            // child process
            close(server_socket); // don't need this
            // handle the client
            client_handler(client_fd);
        } else {
            // server process
            close(client_fd); // don't need this
        }
    }

error:  // fallthrough
    return -1;
}

```

.\ex51b\statserve\src\statserve.h

```c
#ifndef _statserve_h
#define _statserve_h

#include <lcthw/bstrlib.h>
#include <lcthw/ringbuffer.h>
#include <lcthw/stats.h>

struct Command;

typedef int (*handler_cb)(struct Command *cmd, RingBuffer *send_rb, bstring path);

typedef struct Command {
    bstring command;
    bstring name;
    struct bstrList *path;
    bstring number;
    bstring arg;
    handler_cb handler;
} Command;

typedef struct Record {
    bstring name;
    Stats *stat;
} Record;

struct tagbstring OK;

int setup_data_store(const char *store_path);

struct bstrList *parse_name(bstring name);

int scan_paths(Command *cmd, RingBuffer *send_rb);

int parse_line(bstring data, RingBuffer *send_rb);

int run_server(const char *host, const char *port, const char *store_path);

bstring sanitize_location(bstring base, bstring path);

bstring encrypt_armor_name(bstring name);

#endif

```

.\ex51b\statserve\tests\statserve_tests.c

```c
#include "minunit.h"
#include <dlfcn.h>
#include "statserve.h"
#include <lcthw/bstrlib.h>
#include <lcthw/ringbuffer.h>
#include <assert.h>

typedef struct LineTest {
    char *line;
    bstring result;
    char *description;
} LineTest;

int attempt_line(LineTest test)
{
    int rc = -1;
    bstring result = NULL;

    bstring line = bfromcstr(test.line);
    RingBuffer *send_rb = RingBuffer_create(1024);

    rc = parse_line(line, send_rb);
    check(rc == 0, "Failed to parse line.");
    result = RingBuffer_get_all(send_rb);
    check(result != NULL, "Ring buffer empty.");
    check(biseq(result, test.result), "Got the wrong output: %s expected %s",
            bdata(result), bdata(test.result));

    bdestroy(line);
    RingBuffer_destroy(send_rb);
    return 1; // using 1 for tests
error:

    log_err("Failed to process test %s: got %s", test.line, bdata(result));
    if(line) bdestroy(line);
    if(send_rb) RingBuffer_destroy(send_rb);
    return 0;
}

int run_test_lines(LineTest *tests, int count)
{
    int i = 0;

    for(i = 0; i < count; i++) {
        check(attempt_line(tests[i]), "Failed to run %s", tests[i].description);
    }

    return 1;
error:
    return 0;
}

int fake_command(Command *cmd, RingBuffer *send_rb, bstring path)
{
    check(cmd != NULL, "Bad cmd.");
    check(cmd->path != NULL, "Bad path.");
    check(send_rb != NULL, "Bad send_rb.");
    check(path != NULL, "Bad path given.");

    return 0;
error:
    return -1;
}

char *test_path_parsing()
{
    struct bstrList *result = NULL;
    struct tagbstring slash = bsStatic("/");
    struct tagbstring logins_zed = bsStatic("/logins/zed");
    struct tagbstring command_name = bsStatic("dump");
    RingBuffer *send_rb = RingBuffer_create(1024);
    struct bstrList *path = bsplits(&logins_zed, &slash);
    int rc = 0;

    Command fake = {
        .command = &command_name,
        .name = &logins_zed,
        .number = NULL,
        .handler = fake_command,
        .path = path
    };

    result = parse_name(&logins_zed);
    mu_assert(result != NULL, "Failed to parse /logins/zed");

    rc = scan_paths(&fake, send_rb); 
    mu_assert(rc != -1, "scan_paths failed.");

    return NULL;
}

char *test_create()
{
    LineTest tests[] = {
        {.line = "create /zed 100", .result = &OK, .description = "create zed failed"},
        {.line = "create /joe 100", .result = &OK, .description = "create joe failed"},

    };

    mu_assert(run_test_lines(tests, 2), "Failed to run create tests.");

    return NULL;
}

char *test_sample()
{
    struct tagbstring sample1 = bsStatic("100.000000\n");

    LineTest tests[] = {
        {.line = "sample /zed 100", .result = &sample1, .description = "sample zed failed."}
    };

    mu_assert(run_test_lines(tests, 1), "Failed to run sample tests.");

    return NULL;
}

char *test_store_load()
{
    LineTest tests[] = {
        {.line = "delete /zed", .result = &OK, .description = "delete zed failed"},
        {.line = "create /zed 100", .result = &OK, .description = "create zed failed"},
        {.line = "store /zed", .result = &OK, .description = "store zed failed"},
        {.line = "load /zed /sam", .result = &OK, .description = "load zed failed"},
        {.line = "delete /sam", .result = &OK, .description = "load zed failed"},
    };

    mu_assert(run_test_lines(tests, 3), "Failed to run sample tests.");

    return NULL;
}

char *test_encrypt_armor_name()
{
    struct tagbstring test1 = bsStatic("/logins");
    struct tagbstring expect1 = bsStatic("vtmTmzNI");
    struct tagbstring test2 = bsStatic("../../../../../../../../etc/passwd");
    struct tagbstring expect2 = bsStatic("pVOBpFjHEIhB7cuT3BGUvyZGn3lvyj226mgggggg");

    bstring result = encrypt_armor_name(&test1);
    debug("Got encrypted name %s", bdata(result));
    mu_assert(biseq(result, &expect1), "Failed to encrypt test2.");
    bdestroy(result);

    result = encrypt_armor_name(&test2);
    debug("Got encrypted name %s", bdata(result));
    mu_assert(biseq(result, &expect2), "Failed to encrypt test2.");
    bdestroy(result);

    return NULL;
}

char *test_path_sanitize_armor()
{
    struct tagbstring base = bsStatic("/tmp");
    struct tagbstring test1 = bsStatic("/somepath/here/there");
    bstring encname = encrypt_armor_name(&test1);
    bstring expect = bformat("%s/%s", bdata(&base), bdata(encname));
    struct tagbstring test2 = bsStatic("../../../../../../../../etc/passwd");

    bstring result = sanitize_location(&base, &test1);
    mu_assert(result != NULL, "Failed to sanitize path.");
    mu_assert(biseq(result, expect), "failed to sanitize test1");

    // this should be pulled up into a tester function
    // BUG: just get rid of this and use md5
    encname = encrypt_armor_name(&test2);
    expect = bformat("%s/%s", bdata(&base), bdata(encname));
    result = sanitize_location(&base, &test2);
    mu_assert(result != NULL, "Failed to sanitize path.");
    mu_assert(biseq(result, expect), "failed to sanitize test1");

    return NULL;
}

char *all_tests()
{
    mu_suite_start();

    int rc = setup_data_store("/tmp");
    mu_assert(rc == 0, "Failed to setup the data store.");

    mu_run_test(test_path_parsing);
    mu_run_test(test_encrypt_armor_name);
    mu_run_test(test_path_sanitize_armor);
    mu_run_test(test_create);
    mu_run_test(test_sample);
    mu_run_test(test_store_load);

    return NULL;
}

RUN_TESTS(all_tests);

```

Final Review

<file_sep>+++
title = "AWS: EC2 - 1"
description = "Reset EC2 password"
draft="true"
+++


##  EC2



### EC2 with Linux



### EC2 with Windows 



<file_sep>+++
title = "AWS: EKS - 1"
description = "Create a cluster"
weight=11
+++


## EKS - Part 1

Amazon Elastic Kubernetes Service (Amazon EKS) is a managed service that makes it easy for you to run Kubernetes on AWS without needing to stand up or maintain your own Kubernetes control plane. Kubernetes is an open-source system for automating the deployment, scaling, and management of containerized applications.


* EKS runs Kubernetes control plane instances across multiple Availability Zones to ensure high availability.
* EKS automatically detects and replaces unhealthy control plane instances.
* EKS provides automated version upgrades and patching for them.
* EKS is also integrated with many AWS services to provide scalability and security.




### eksctl

* Install the Latest AWS CLI

        pip install awscli --upgrade --user

* Install eksctl (Mac)

```
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

#Install the Weaveworks Homebrew tap.
brew tap weaveworks/tap

# Install or upgrade eksctl.
brew install weaveworks/tap/eksctl

brew upgrade eksctl && brew link --overwrite eksctl

eksctl version
```

* Install eksctl (Linux)

```
# The latest version is 0.16
curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp

## For EKS with workloads   0.17.0-rc.0 is required
# curl --silent --location "https://github.com/weaveworks/eksctl/releases/download/0.17.0-rc.0/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp


#Install the Weaveworks Homebrew tap.
sudo mv /tmp/eksctl /usr/local/bin

eksctl version
```

### EKS with Fargate

#### Fargate

AWS Fargate is a serverless compute engine for containers that works with both Amazon Elastic Container Service (ECS) and Amazon Elastic Kubernetes Service (EKS). Fargate makes it easy for you to focus on building your applications. Fargate removes the need to provision and manage servers, lets you specify and pay for resources per application, and improves security through application isolation by design.

* Deploy and manage applications, not infrastructure
* Right-sized resources with flexible pricing options
* Secure isolation by design
* Rich observability of applications

##### Pod Configuration

vCPU value | Memory value
------|------
.25 vCPU| 0.5 GB, 1 GB, 2 GB
.5 vCPU| 1 GB, 2 GB, 3 GB, 4 GB
1 vCPU| 2 GB, 3 GB, 4 GB, 5 GB, 6 GB, 7 GB, 8 GB
2 vCPU | Between 4 GB and 16 GB in 1-GB increments
4 vCPU | Between 8 GB and 30 GB in 1-GB increments



#### Create Cluster

```
CLUSTER_NAME="pg-prd"
REGION_CODE="ap-southeast-1"

eksctl create cluster \
--name ${CLUSTER_NAME} \
--region ${REGION_CODE} \
--fargate
```

### EKS with EC2



```
CLUSTER_NAME="pg-prd"
REGION_CODE="ap-southeast-2"
NODE_GRP_NAME="standard-workers"
KEY_NAME="nonprod-kp"
NODE_TYPE="t3.medium"

eksctl create cluster \
--name  ${CLUSTER_NAME}  \
--region ${REGION_CODE} \
--nodegroup-name ${NODE_GRP_NAME} \
--node-type ${NODE_TYPE} \
--nodes 1 \
--nodes-min 1 \
--nodes-max 3 \
--ssh-access \
--ssh-public-key "${KEY_NAME}" \
--managed
```


#### Delete Cluster

* Get all services

        kubectl get svc --all-namespaces

* Delete all services with EXTERNAL-IP 

        kubectl delete svc <service-name>

* Delete cluster

        eksctl delete cluster --name <cluster-name>


<file_sep>+++

date = "2016-08-11T11:59:31+11:00"
title = "PHP Web Framework"
description="Introduction of PHP Web Frameworks:  Zend Framework, Laravel "

draft = false
+++

> *Here we are going to explore some PHP web frameworks.*

## PHP development environment setup

### Install PHP 5.6.x

* Please find the instruction from [home page](/#php)

* Composer 

### Linux 

* Use `curl -s https://getcomposer.org/installer | php --` to install composer on Linux 
* use 'composer -v ' to verify.

### Windows

* Download the [composer](https://getcomposer.org/download/) and install php on your PC
* Use `composer -v` to verify the composer is ready.


## Zend Framework

> Zend Framework 2.x is a collection of 60+ packages for professional PHP development. It can be used to develop web applications and services using PHP 5.6+, and provides 100% object-oriented code using a broad spectrum of language features.


### Create Zend Framework 2 project 

    * Clone Zend Framework skeleton project as new project. 

    * Install zend framework with composer



```bash
    cd  /path/to/newproject
    git clone git://github.com/zend framework/ZendSkeletonApplication.git
    cd ZendSkeletonApplication
    php composer.phar self-update
    php composer.phar install
```


### Start app with php built-in server 

**Linux**

```bash
php -S 0.0.0.0:8080 -t public/ public/index.php
```

**Windows**
```bash
php -S 0.0.0.0:8080 -t public public/index.php
```

### Use apache server

* Apache configuration

```apache
 <VirtualHost *:80>
     ServerName zf2-tutorial.localhost
     DocumentRoot /path/to/newproject/ZendSkeletonApplication/public
     SetEnv APPLICATION_ENV "development"
     <Directory /path/to/newproject/ZendSkeletonApplication/public>
         DirectoryIndex index.php
         AllowOverride All
         Require all granted
     </Directory>
 </VirtualHost>
```

* Zf2-MVC-Starter project
 
 * Please find the project introduction [here](/projects/zf2-mvc-starter/). 

## Laravel

> Laravel is a free, open-source PHP web framework, created by <NAME> and intended for the development of web applications following the model–view–controller (MVC) architectural pattern. Some of the features of Laravel are a modular packaging system with a dedicated dependency manager, different ways for accessing relational databases, utilities that aid in application deployment and maintenance, and its orientation toward syntactic sugar.


### Install laravel global 
   
    * Use `composer global laravel/installer`

    * Enable the `mbs-string` extension

    * update `php.ini` config 
    
    * Open `php.ini` with notepad
    
    * Change `;extension=php_mbstring.dll` to `extension=php_mbstring.dll`


### Create new project from scratch

#### Migrate database and seed dummy data

* Create data model
* Following is the sample code 

```php 
<?php

namespace App;
use Illuminate\Database\Eloquent\Model;

class Article extends Model
{
    protected $fillable = ['title', 'content'];

    public function getTitleAttribute($value) {
        return strtoupper($value);
    }
}
```
* Use artisan to create the table 

```bash

php artisan migrate -VVV 

## Use following command to seek dump or initial data
php artisan db:seed
```

### Troubleshooting

**Fix the error of Specified key was too long**

```php
namespace App\Providers;

use Illuminate\Support\ServiceProvider;
use Illuminate\Support\Facades\Schema;

class AppServiceProvider extends ServiceProvider
{
    /**
     * Bootstrap any application services.
     *
     * @return void
     */
    public function boot()
    {
        //
        Schema::defaultStringLength(191);
    }

    /**
     * Register any application services.
     *
     * @return void
     */
    public function register()
    {
        //
    }
}
```

### Lavarel MVC Starter project 
   
    * Please find the project introduction [here](/projects/lara-mvc-starter/). 

## Use Laravel Rest Starter project

## PrestaShop 

> PrestaShop is a free, open source e-commerce solution. The software is published under the Open Software License (OSL). It is written in the PHP programming language with support for the MySQL database management system.


### Download the zip file from [download page](https://www.prestashop.com/en/developers-versions)

### Install Prestashop

There is a instruction page inside the zip file. You can follow the instructions to complete the installation. There is no EasyPHP, Wamp, XAMPP, or any similar AMP (Apache+MySQL+PHP) package installed on my PC, but I have PHP, Apache, MySQL installed. Actually EasyPHP, Wamp are just the bundle of PHP development tools, which include PHP, Apache, MySQL. I don't want to install too many duplicate softwares and packages on my PC, so I prefer to install Prestashop with what I have on my PC. Which strategy is up to you.

### Install Prestashop with AMP package

* Follow the instruction page within zip file.

### Install Prestashop without AMP package

- Unzip file to `path\to\prestashop_workspace`. Your folder structure will look like this. 

```ini
path\to\prestashop_workspace
\---prestashop
    +---admin 
    +---cache 
    +---classes  
    +---config 
    +---controllers 
    +---css
    +---docs
    +---download
    +---img 
    +---install    
    +---js   
    +---localization
    +---log
    +---mails
    +---modules  
    +---override
    +---pdf
    +---themes
    +---tools 
    +---translations
    +---upload
    \---webservice
```

* Start your MySQL or check the status of MySQL
    * Use your MySQL client tool to connect to your MySQL server. 


* Launch installer page with php server
* Start a command prompt

```bash
cd /path/to/prestashop_workspace
php -S 0.0.0.0:1234 -t prestashop
```

* Open the link `http://localhost:1234/install/index.php` with browser, then you can start installation process.

* Choose language and click `Next`, and then select the checkbox "I agree bah lah bah lah .... " and click `Next`

* If there is an error `GD Library is not installed` prompt, you just need to enable the library on `php.ini`
  * DO NOT close your browser. 
  * Stop the php server by `Ctrl + C` in the command prompt.
  * Use notepad to open the file `php.ini` under the `\path\to\php`
  * Uncomment the config `;extension=php_gd2.dll` => `extension=php_gd2.dll`
  * Start the php server again
  
* Click the `Refresh this settings`, and click `Next` 
* Fill the login user and password. If your MySQL port is not 3306, please attach your port to the server address input field. Click `Test your database connection`. 

* If you got error `prestashop database not found`, you need to create a database on mysql server.
    * I simply create a new database immediately with one command line

    ```sql
    CREATE DATABASE prestashop CHAR SET utf8 COLLATE 'utf8_unicode_ci';
    ```
   
    * Test the connection again. You will get the green light
    
*  Click `Next` and you can start to setup your store informaiton, such as, store name, admin account, etc. Then click `Next`

* Setup your sample store. Click `Next`. Then the installer will help you finish the initialization.

* After the store setup, you can access the website by clicking `Font site`, but you can not access back office, as known as admin panel. 

* Don't panic. It is easy to fix. Stop the php server by clicking `Ctrl+C`, and delete the folder `install` under the root, and then start the server again. Open the folder with prestashop, you will find something interesting. The original folder `admin` under `prestashop` has been renamed to `adminXXXX`. X is a number. It is Prestashop special trick to secure your admin folder. Now you need to use this new name as path to acces back office. Your new back office link will be `http://localhost:1234/adminXXXX`. 

* Open the new link in browser and type in your admin id and password. Now you can start managing your Prestashop site. Enjoy it.   

### Forgot admin's password

* Forgot admin's password or somehow you have to reset password and you cannot get admin's password from previous adminstrator. For such case, there is a simple way to update admin's password from database. 

* Tailor the SQL below. Then you should be able to use new password to login. 

```sql
UPDATE ps_employee SET passwd = MD5('<_COOKIE_KEY_><PASSWORD>') 
WHERE ps_employee.id_employee = <ID_EMPLOYEE>;
```

### Troubleshooting

#### InnoDB error

* Error : InnoDB is not supported by your MySQL server 

> You got error because Prestashop 1.5 is not working properly with MySQL 5.4 and later. 

* Update files `DbPDO.php` , `MySQL.php` and `DbMySQLi.php` as follow.

```php
    // $sql = 'SHOW VARIABLES WHERE Variable_name = \'have_innodb\'';
    $sql = "SELECT SUPPORT FROM INFORMATION_SCHEMA.ENGINES WHERE ENGINE LIKE 'INNO%'";

    ...
    
    // if (!$row || strtolower($row['Value']) != 'yes')
    if (!$row || strtolower($row['Value']) === 'no') 
``` 

* Restart the server and the proble will be fixed.


#### CORS 

* Enable module header in `httpd.conf`

* Add header settings

```apache
Header always set Access-Control-Max-Age "1000"

#  "X-Requested-With, Content-Type, Origin, Authorization, Accept, Client-Security-Token, Accept-Encoding"
Header always set Access-Control-Allow-Headers  "USE ABOVE COMMENT AS DEFAULT VALUE"
Header always set Access-Control-Allow-Methods "POST, GET, OPTIONS, DELETE, PUT"
```

#### Too big header file 

* Add this to your `http {}` of the nginx.conf file normally located at `/etc/nginx/nginx.conf`:

    proxy_buffer_size   128k;
    proxy_buffers   4 256k;
    proxy_busy_buffers_size   256k;


* Then add this to your php location block, this will be located in your vhost file look for the block that begins with `location ~ .php$ {`

    fastcgi_buffer_size 128k;
    fastcgi_buffers 4 256k;
    fastcgi_busy_buffers_size 256k;



<file_sep>+++
title = "Azure: App - 1"
weight = 3
description="Introduction of Azure App Service"
+++

## Azure App Service


Azure App Service is an HTTP-based service for hosting web applications, REST APIs, and mobile back ends. You can develop in your favorite language, be it .NET, .NET Core, Java, Ruby, Node.js, PHP, or Python. Applications run and scale with ease on both Windows and Linux-based environments.

* Built-in auto scale support

* Continuous integration/deployment support

* Deployment slots


### Limitations

* App Service on Linux is not supported on Shared pricing tier.
* You can't mix Windows and Linux apps in the same App Service plan.
* The Azure portal shows only features that currently work for Linux apps. 


### Create a Web App 

#### Create a web app via docker container

* Set defaout subscriptoin

```
az account set --subscription XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX

```

* Crate a resource group

```
az group create --name my-ResourceGroup --location <your-region>

```

* Create a service plan
  
> By default, the command below uses an inexpensive B1 pricing tier that is free for the first month. You can control the tier with the --sku parameter.

```
az appservice plan create --name myAppServicePlan \
    --resource-group myResourceGroup --is-linux
```


```sh
az webapp create --resource-group myAppServicePlan \
     --plan myAppServicePlan --name  myApp  \
     --deployment-container-image-name XXXXX/my-web-app:latest
```


#### Add Lets Encrypt SSL for custom domain or sub-domain

> ___The approach below is only for experiment. It is not scalable for commercial purpose.___

* Add customized sub-domain to your DNS

```sh
    sub-domain.domain.com.	1800	IN	A	172.16.58.3
```


* Generate the SSL cert via Certbot. Following example is done on a linux machine, which IP is `10.10.10.10`

```sh
    sudo certbot  --nginx -d sub-domain.domain.com
```

* Generate the pfx file from pem file

> __Remember the password of the SSL cert__

```sh
    cd
    sudo openssl pkcs12 -inkey /etc/letsencrypt/live/sub-domain.domain.com/privkey.pem  -in /etc/letsencrypt/live/sub-domain.domain.com/fullchain.pem -in -export  -out sub-domain.domain.com.pfx
    Enter Export Password:
    Verifying - Enter Export Password:
```

* Download the certificate file 

```sh
    scp -i ~/.ssh/XXX_rsa username@10.10.10.10:/sub-domain.domain.com.pfx ~/
```



* Follow the Azure App Service instruction to add customized domain

    * Navigate to `Custom Domain` page: App Service > Settings > custom domains
    * __Remove the previous A Record of this sub-domain from your DNS__
    * Add following records in your DNS 
    
    ```
    sub-domain.domain.com. 1800	      IN	CNAME	sub-domain.azurewebsites.net.
    asuid.sub-domain.domain.com. 1800 IN	TXT  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    ```

    * Validate the custom domain 
    * Add the custom domain

* Add the SSL to custom domain 

    * Under the SSL session there is alarm sign which shows the domain has no SSL certiificate 
    * Click `Add binding`
    * Upload the pfx certificate file generated from previous step
    * Enter the password of the certificate 
    * Select default option for `Private Certificate Thumbprint` (Only one option available)
    * Select  `SNI SSL` for TSL Type  (Only one option available)

* Check the SSL with SSL online checker
  

<file_sep>+++
title = " Generic Predicate & Expression"
description = "Use generic predicate and expression for data query"
+++




### Generic Predicate and Expression for query

#### Prerequisites

> *You have experience of developing .net application, which includes entity framework.*

> *You have experience of using SQL to query database*

#### Problem

Now entity framework is core component in all .net applications, which need to communicate with database. 

Business service get a lot of benefits form entity framework's ORM feature, and we can create a repository layer on the top of ORM to reduce some simple but tedious database operation, such as, delete, insert, query all data. However, when we need to do some complicated query to support business service, we still need to take so much effort to achieve the query result, because entity framework use LINQ as query language, comparing with SQL, native database language, it is a bit more complicated and cumbersome. Luckily, entity framework provide another generic feature to help us DRY. Predicate and expression can help us create generic query and reduce many reduntant code. 


#### Solution

### Analysis

Basically, the idea is close to dynamic statement in ADO.Net. Here generic programming is the key, when we utilize predicate and expression to dynamically rebuild the query filter.  

If we look into the queries, we will find 80% of queries can be abstracted as following syntax. Now it is easy to see how generic predicate and expression can support this syntax.    

```sql
SELECT * FROM TABLE_A 
    WHERE 
        <FIELD_1> [=|<|>|>=|<=|LIKE] <VALUE_1>   ----- Expression 
        [ AND | OR ]                            ------------ Predicate 
        <FIELD_2> [=|<|>|>=|<=|LIKE] <VALUE_2>   ----- Expression
``` 

The next step we can look into the Expression, actually the FIELD_1 is the property of entity object, and VALUE_1 is the filter value entered by client. How to use the filter to narrow down the query result is part of business logic, which is handled by developer. 

```ini 
<FIELD_1>           ==> Entity property
[=|<|>|>=|<=|LIKE]  ==> Operator
<VALUE_1>           ==> Filter value
```

### Design

Accordingt to abve analysis, we can design the classes to support this feature.

```ini

+--------------+
|  Filter      |
+--------------+
|  Property    | --- Map to the column (table) or property (entity)
|  Op          | --- Operator , e.g. Equals, Contains, GreaterThan, tec.
|  Val         | --- Value entered by client
+--------------+

+----------------------+
| ExpressionBuilder    |  
+----------------------+
|GetExpression( filer) | --------- Create expression by input filter 
+ ---------------------+

+------------------+
| PredicateBuilder |
+------------------+      
| And(exp1, exp2)  | --- Use AND to combine two expressions
| Or(exp1, exp2)   | --- Use OR to combine two expressions                        
+------------------+

```

### Implementatoin

Following is the implementation of generic expression and predicate. Please keep it in mind. We are using LINQ to simulate the dynamic statement, so there is some tricks to work around as SQL queries. 

**Filter**

The Op property should be 

```cs
public class Filter
{
    public string Property { get; set; }
    public string Op { get; set; }
    public object Val { get; set; }
}
```

**Op**

This class can be replaced by Enum if you want.  


```cs

public static class Op
{
    public const string Equals = "Equals";
    public const string GreaterThan = "GreaterThan";
    public const string LessThan = "LessThan";
    public const string GreaterThanOrEqual = "GreaterThanOrEqual";
    public const string LessThanOrEqual = "LessThanOrEqual";

    public const string Contains = "Contains";
    public const string StartsWith = "StartsWith";
    public const string EndsWith = "EndsWith";
}
```


**ExpressionBuilder**

This class takes care of Expression with Filter object.

```cs
using System;
using System.Linq;
using System.Linq.Expressions;
using System.Collections.Generic;

public class ExpressionBuilder
{
    private static MethodInfo containsMethod = typeof(string).GetMethod("Contains");
    private static MethodInfo startsWithMethod =
    typeof(string).GetMethod("StartsWith", new Type[] { typeof(string) });
    private static MethodInfo endsWithMethod =
    typeof(string).GetMethod("EndsWith", new Type[] { typeof(string) });


    public static Expression<Func<T, bool>> GetExpression<T>(IList<Filter> filters)
    {
        if (filters.Count == 0)
            return null;

        ParameterExpression param = Expression.Parameter(typeof(T), "t");
        Expression exp = null;

        if (filters.Count == 1)
            exp = GetExpression<T>(param, filters[0]);
        else if (filters.Count == 2)
            exp = GetExpression<T>(param, filters[0], filters[1]);
        else
        {
            while (filters.Count > 0)
            {
                var f1 = filters[0];
                var f2 = filters[1];

                if (exp == null)
                    exp = GetExpression<T>(param, filters[0], filters[1]);
                else
                    exp = Expression.AndAlso(
                        exp, 
                        GetExpression<T>(param, filters[0], filters[1]));

                filters.Remove(f1);
                filters.Remove(f2);

                if (filters.Count == 1)
                {
                    exp = Expression.AndAlso(
                        exp, 
                        GetExpression<T>(param, filters[0]));
                    filters.RemoveAt(0);
                }
            }
        }

        return Expression.Lambda<Func<T, bool>>(exp, param);
    }

    private static ConstantExpression ConvetValueType(
        MemberExpression member, 
        object value)
    {
        if (member.Type == typeof(int))
        {
            value = int.Parse(value.ToString());
        }
        else if (member.Type == typeof(decimal))
        {
            value = decimal.Parse(value.ToString());
        }
        else if (member.Type == typeof(float))
        {
            value = float.Parse(value.ToString());
        }
        else if (member.Type == typeof(double))
        {
            value = double.Parse(value.ToString());
        }

        return Expression.Constant(value);
    }

    private static Expression GetExpression<T>(
        ParameterExpression param, 
        Filter filter)
    {

        MemberExpression member = Expression.Property(
            param, filter.Property);

        switch (filter.Op)
        {
            case Op.Equals:
                return Expression.Equal(
                    member, 
                    Expression.Constant(filter.Val, member.Type));

            case Op.GreaterThan:
                return Expression.GreaterThan(
                    member, 
                    ConvetValueType(member, filter.Val));

            case Op.GreaterThanOrEqual:
                return Expression.GreaterThanOrEqual(
                    member, 
                    ConvetValueType(member, filter.Val));

            case Op.LessThan:
                return Expression.LessThan(
                    member, 
                    ConvetValueType(member, filter.Val));

            case Op.LessThanOrEqual:
                return Expression.LessThanOrEqual(
                    member, 
                    ConvetValueType(member, filter.Val));

            case Op.Contains:
                return Expression.Call(
                    member, 
                    containsMethod, 
                    Expression.Constant(filter.Val, member.Type));

            case Op.StartsWith:
                return Expression.Call(
                    member, 
                    startsWithMethod, Expression.Constant(
                        filter.Val, 
                        member.Type));

            case Op.EndsWith:
                return Expression.Call(
                    member, 
                    endsWithMethod, 
                    Expression.Constant(filter.Val, member.Type));
        }

        return null;
    }

    private static BinaryExpression GetExpression<T>(\
        ParameterExpression param, Filter filter1, Filter filter2)
    {
        Expression bin1 = GetExpression<T>(param, filter1);
        Expression bin2 = GetExpression<T>(param, filter2);

        return Expression.AndAlso(bin1, bin2);
    }
}

```

**PredicateBuilder**

This class manages all expressions to support dynamic statement query.

```cs 
using System;
using System.Linq;
using System.Linq.Expressions;
using System.Collections.Generic;

public static class PredicateBuilder
{
    public static Expression<Func<T, bool>> True<T>() { return f => true; }
    public static Expression<Func<T, bool>> False<T>() { return f => false; }

    public static Expression<Func<T, bool>> Or<T>(
        this Expression<Func<T, bool>> expr1, Expression<Func<T, bool>> expr2)
    {
        var secondBody = expr2.Body.Replace(
            expr2.Parameters[0], expr1.Parameters[0]);
        return Expression.Lambda<Func<T, bool>>(
            Expression.OrElse(expr1.Body, secondBody), 
            expr1.Parameters);
    }

    public static Expression<Func<T, bool>> And<T>(
        this Expression<Func<T, bool>> expr1, 
        Expression<Func<T, bool>> expr2)
    {
        var secondBody = expr2.Body.Replace(
            expr2.Parameters[0], expr1.Parameters[0]);
        return Expression.Lambda<Func<T, bool>> (
                    Expression.AndAlso(expr1.Body, secondBody), 
                    expr1.Parameters);
    }

    public static Expression Replace(
        this Expression expression, 
        Expression searchEx, 
        Expression replaceEx)
    {
        return new ReplaceVisitor(searchEx, replaceEx).Visit(expression);
    }

    internal class ReplaceVisitor : ExpressionVisitor
    {
        private readonly Expression from, to;
        public ReplaceVisitor(Expression from, Expression to)
        {
            this.from = from;
            this.to = to;
        }

        public override Expression Visit(Expression node)
        {
            return node == from ? to : base.Visit(node);
        }
    }
}
```<file_sep>+++
title = "AWS: VPC - 4"
description = "VPC - Simple Demo "
weight=6

+++


## VPC Part 4




### Simples demo


* Diagram of customized VPC - MyDemoVPC with Internet Gatway and VPN connect

{{<mermaid>}}
graph LR
    InternetGW(Internet Gateway)
    VirtualGW(Virtual Gateway)
    INTER(Internet - Public)
    InternetGW --- INTER
    VirtualGW --- SERVER
    subgraph MyDemoVPC
        EC2_A(EC2 Instannce A)
        EC2_B(EC2 Instannce B)
        EC2_E(EC2 Instannce E)
        EC2_F(EC2 Instannce F)
        EC2_C[(Database Master)]
        EC2_D[(Database Slave)]
        MainRouteTable(10.0.0.0/16)
        PrvSubnet(10.0.2.0/24)
        PubSubnet(10.0.1.0/24)
        VPNSubnet(10.0.3.0/24)
        MainRouteTable --- InternetGW
        MainRouteTable --- NetworkACL
        NetworkACL --- PubSecGrp
        NetworkACL --- PrivSecGrp
        PrivSecGrp --- PrvSubnet
        VPNSubnet --- VirtualGW
        VPNSubnet --- MainRouteTable
        PubSecGrp --- PubSubnet
        subgraph Implied_Router
            MainRouteTable(10.0.0.0/16)
        end 
        subgraph Private_Subnet
             PrvSubnet
             EC2_C
             EC2_D
        end
        subgraph Public_Subnet
            PubSubnet
            EC2_A
            EC2_B
        end
        subgraph VPN_Subnet
             VPNSubnet
             EC2_E
             EC2_F
        end
    end
    InternetGW
    subgraph Internet
        INTER
    end
    subgraph OnPremise
        SERVER
    end


{{</mermaid>}}



* Customized Route tables of Subnet Public_Subnet

Destination | Target 
----- | ------- 
10.0.1.0/16 | local 
fd00:a516:7c1b:17cd:6d81:2137:bd2a:2c5b/56 | local 
0.0.0.0/0 | InternetGW 
::0/0 | InternetGW 


* Main Route tables of Subnet Private_Subnet

Destination | Target
------------|---------
10.0.1.0/16 | local
fd00:a516:7c1b:17cd:6d81:2137:bd2a:2c5b/56 | local


* Route table of Subnet VPN_Subnet

Destination | Target
------------|---------
10.0.1.0/16 | local
fd00:a516:7c1b:17cd:6d81:2137:bd2a:2c5b/56 | local
0.0.0.0/0 | VirtualGW
<file_sep>+++
title="Goroutine & Channel - 2"
description="Goroutine & Channels - Part 2"
weight = 7
+++



### Channel directionality

* A channel type may be annotated to specify that it may only send or only receive

        var send_only chan<- int // channel can only receive data
        var recv_only <-chan int // channel can only send data

* Receive-only channels ( <-chan T ) cannot be closed, because closing a channel is intended as a way for a sender to signal that no more values will be sent to the channel, so it has no meaning for receive-only channels.




#### Pipe & Filter pattern

* A goroutine channel process will processes what it receives from an input channel and sends this to an output channel


    ```go
    func main() {
        sendChan := make(chan int)
        receiveChan := make(chan string)
        go processChannel(sendChan, receiveChan)

    }

    func processChannel(in <-chan int, out chan<- string) {
        for inValue := range in {
            result := strconv.Itoa(inValue) // processing inValue
            // ...
            out <- result
        }
    }
    ```

* Prime number generator sample

    ```go
    func generate() chan int {
        ch := make(chan int)
        go func() {
            for i := 2; ; i++ {
                ch <- i
            }
        }()
        return ch
    }

    // Filter out input values divisible by prime, send rest to returned channel
    func filter(in chan int, prime int) chan int {
        out := make(chan int)
        go func() {
            for {
                if i := <-in; i%prime != 0 {
                    out <- i
                }
            }
        }()
        return out
    }

    func sieve() chan int {
        out := make(chan int)
        go func() {
            ch := generate()
            for {
                prime := <-ch
                ch = filter(ch, prime)
                out <- prime
            }
        }()
        return out
    }

    func main() {
        primes := sieve()
        for {
            fmt.Println(<-primes)
        }
    }
    ```

##### Goroutine with select

* The select chooses which of the multiple communications listed by its cases can proceed. The default clause is optional; fall through behavior, like in the normal switch, is not permitted.

    * It all are blocked, it waits until one can proceed.
    * If multiple can proceed, it chooses one at random.
    * When none of the channel operations can proceed and the default clause is present, then this is executed: the default is always runnable (that is: ready to execute).

* Using a send operation in a select statement with a default case guarantees that the send will be non-blocking! If there are no cases, the select blocks execution forever.

* Sample 9

    ```go
    func main() {
        runtime.GOMAXPROCS(2) // in goroutine_select2.go
        ch1 := make(chan int)
        ch2 := make(chan int)
        go pump1(ch1)
        go pump2(ch2)
        go suck(ch1, ch2)
        time.Sleep(1e9)
    }
    func pump1(ch chan int) {
        for i := 0; ; i++ {
            ch <- i * 2
        }
    }
    func pump2(ch chan int) {
        for i := 0; ; i++ {
            ch <- i + 5
        }
    }
    func suck(ch1 chan int, ch2 chan int) {
        for {
            select {
            case v := <-ch1:

                fmt.Printf("Received on channel 1: %d\n", v)
            case v := <-ch2:
                fmt.Printf("Received on channel 2: %d\n", v)
            }
        }
    }
    ```


##### Timeout & Ticker

* Ticker: a struct time.Ticker which is an object that repeatedly sends a time value on a contained channel C at a specified time interval


    ```go
    type Ticker struct {
        C <-chan Time // the channel on which the ticks are delivered.
                    // contains filtered or unexported fields
        // ...
    }
    ```


* A ticker is stopped with Stop(), use this in a defer statement.

    ```go
    ticker := time.NewTicker(updateInterval)
    defer ticker.Stop()
    // ...
    select {
        case u:= <- ch1:
            // ...
        case v:= <- ch2:
            // ...
        case <- ticker.C:
            logState(status) // call some logging function logState
        default: // no value ready to be received
            // ...
    }
    ```

* Handy to use when you have to limit the rate of processing per unit time. The time.Tick() function with signature func Tick(d Duration) <-chan Time is useful when you only need access to the return channel and don’t need to shutdown it.

* Sample below is good use case for time.Tick function

    ```go
    rate_per_sec := 10
    var dur Duration = 1e9 / rate_per_sec
    chRate := time.Tick(dur) // a tick every 1/10th of a second
    for req := range requests {
        <- chRate // rate limit our Service.Method RPC calls
        go client.Call("Service.Method", req, ...) // client.Call is RPC call
    }
    ```


* A Timer type looks exactly the same as a Ticker type (it is constructed with NewTimer(d but it sends the time only once, after a Duration d.

* After Duration d the current time is sent on the returned channel; so this is equivalent to NewTimer(d).C; it resembles Tick(), but After() sends the time only once.

* Sample of timer 

    ```go
    func main() {
        tick := time.Tick(1e8)
        boom := time.After(5e8)
        for {
            select {
                case <-tick:
                    fmt.Println("tick.")
                case <-boom:
                    fmt.Println("BOOM!")
                    return
                default:
                    fmt.Println(" .")
                    time.Sleep(5e7)
            }
        }
    }
    ```


### Generator

#### Lazy generator

* A generator is a function that returns the next value in a sequence each time the function is called.

* It is a producer that only returns the next value, not the entire sequence; this is called lazy evaluation:only compute what you need at the moment, saving valuable resources (memory and CPU): it is a technology for the evaluation of expressions on demand.

##### Basic lazy generator


```go
var resume chan int

func integers() chan int {
    yield := make(chan int)
    count := 0
    go func() {
        for {
            yield <- count
            count++
        }
    }()
    return yield
}
func generateInteger() int {
    return <-resume
}
func main() {
    resume = integers()
    fmt.Println(generateInteger()) //=> 0
    fmt.Println(generateInteger()) //=> 1
    fmt.Println(generateInteger()) //=> 2
}
```

##### Generic Lazy Generator

* By making clever use of the empty interface, closures and higher order functions we can implement a generic builder BuildLazyEvaluator for the lazy evaluation function (this should best placed inside a utility package). The builder takes a function that has to be evaluated and an initial state as arguments and returns a function without arguments returning the desired value. The passed evaluation function has to calculate the next return value as well as the next state based on the state argument. Inside the builder a channel and a goroutine with an endless loop are created. The return values are passed to the channel from which they are fetched by the returned function for later usage. Each time a value is fetched the next one will be calculated.


    ```go
    type Any interface{}
    type EvalFunc func(Any) (Any, Any)

    func main() {
        evenFunc := func(state Any) (Any, Any) {
            oldSate := state.(int)
            newState := oldSate + 2
            return oldSate, newState
        }
        even := BuildLazyIntEvaluator(evenFunc, 0)
        for i := 0; i < 10; i++ {
            fmt.Printf("%vth even: %v\n", i, even())
        }
    }

    func BuildLazyEvaluator(evalFunc EvalFunc, initState Any) func() Any {
        retValChan := make(chan Any)
        loopFunc := func() {
            var actState Any = initState
            var retVal Any
            for {
                retVal, actState = evalFunc(actState)
                retValChan <- retVal
            }
        }
        retFunc := func() Any {
            return <-retValChan
        }
        go loopFunc()
        return retFunc
    }

    func BuildLazyIntEvaluator(evalFunc EvalFunc, initState Any) func() int {
        evalFn := BuildLazyEvaluator(evalFunc, initState)
        return func() int {
            return evalFn().(int)
        }
    }
    ```

### Future

> A related idea is that of futures: sometimes you know you need to compute a value before you need to actually use the value. In this case, you can potentially start computing the value on another processor and have it ready when you need it.

* Futures are easy to implement via closures and goroutines, the idea is similar to generators, except
a future needs only to return one value.

* Matrix package will look like the code below

    ```go
    // futures used internally
    type futureMatrix chan Matrix;

    // API remains the same
    func Inverse (a Matrix) Matrix {
        return <-InverseAsync(promise(a))
    }

    func Product (a Matrix, b Matrix) Matrix {
        return <-ProductAsync(promise(a), promise(b))
    }

    // expose async version of the API
    func InverseAsync (a futureMatrix) futureMatrix {
        c := make (futureMatrix)
        go func () { c <- inverse(<-a) } ()
        return c
    }

    func ProductAsync (a futureMatrix) futureMatrix {
        c := make (futureMatrix)
        go func () { c <- product(<-a) } ()
        return c
    }

    // actual implementation is the same as before
    func product (a Matrix, b Matrix) Matrix {
        ....
    }

    func inverse (a Matrix) Matrix {
        ....
    }

    // utility fxn: create a futureMatrix from a given matrix
    func promise (a Matrix) futureMatrix {
        future := make (futureMatrix, 1)
        future <- a;
        return future;
    }
    ```

* Use the matrix package 

    ```go
    func InverseProduct (a Matrix, b Matrix) {
        a_inv := Inverse(a)
        b_inv := Inverse(b)
        return Product(a_inv, b_inv)
    }

    // async way
    func InverseProduct (a Matrix, b Matrix) {
        a_inv_future := InverseAsync(a);
        b_inv_future := InverseAsync(b);
        a_inv := <-a_inv_future;
        b_inv := <-b_inv_future;
        return Product(a_inv, b_inv);
    }
    ```


#### Multiplexing 

* Client-server applications are the kind of applications where goroutines and channels shine.

* Server side simulator sample

    ```go
    type Request struct {
        a, b      int
        replyChan chan int // reply channel inside the Request
    }

    type binOp func(a, b int) int

    func run(op binOp, req *Request) {
        req.replyChan <- op(req.a, req.b)
    }

    func server(op binOp, service chan *Request, quitChan chan bool) {
        for {
            select {
            case req := <-service:
                go run(op, req)
            case <-quitChan:
                return
            }
        }
    }
    func startServer(op binOp) (service chan *Request, quitChan chan bool) {
        service = make(chan *Request)
        quitChan = make(chan bool)
        go server(op, service, quitChan)
        return service, quitChan
    }

    func main() {
        adder, quitChan := startServer(func(a, b int) int { return a + b })
        const N = 100
        var reqs [N]Request
        for i := 0; i < N; i++ {
            req := &reqs[i]
            req.a = i
            req.b = i + N
            req.replyChan = make(chan int)
            adder <- req
        }
        // checks:
        for i := N - 1; i >= 0; i-- { // doesn’t matter what order
            if <-reqs[i].replyChan != N+2*i {
                fmt.Println("fail at", i)
            } else {

                fmt.Println("Request ", i, "is ok!")
            }
        }
        quitChan <- true
        fmt.Println("done")
    }
    ```

### Parallel For-Loop

* In summary, we need a channel for synchronization purposes (used as a semaphore) when implementing a parallel for-loop, but we do not need to communicate with goroutine through channels when the stack works perfectly well.

    ```go
    xi := make(float chan);
    out := make(float chan);
    for _,xi := range data {
        xch := make(float chan);
        go func () {
            xi := <- xch;
            out <- doSomething(xi);
        }()
        xch <- xi;
    }
    ```

* Another sample of parallel-loop with semaphore

    ```go
    func VectorScalarAdd (v []float, s float) {
        sem := make (semaphore, len(v));
        for i,_ := range v {
            go func (i int) {
                v [i] += s; 
                sem.Signal();
            } (i);
        }()
        sem.Wait(len(v));
    }
    ```



#### Concurrent access to object with channel.

* To safeguard concurrent modifications of an object instead of using locking with a sync Mutex we can also use a backend goroutine for the sequential execution of anonymous functions.

* In the following program we have a type Person which now contains a field chF, a channel of anonymous functions. This is initialized in the constructor-method NewPerson, which also starts a method backend() as a goroutine.This method executes in an infinite loop all the functions placed on chF, effectively serializing them and thus providing safe concurrent access. The methods that change and retrieve the salary make an anonymous function which does that and put this function on chF, and backend() will sequentially execute them.


    ```go
    type Person struct {
        Name   string
        salary float64
        chF    chan func()
    }

    func NewPerson(name string, salary float64) *Person {
        p := &Person{name, salary, make(chan func())}
        go p.backend()
        return p
    }
    func (p *Person) backend() {
        for f := range p.chF {
            f()
        }
    }

    // Set salary.
    func (p *Person) SetSalary(sal float64) {
        p.chF <- func() { p.salary = sal }
    }

    // Retrieve salary.
    func (p *Person) Salary() float64 {
        fChan := make(chan float64)
        p.chF <- func() { fChan <- p.salary }
        return <-fChan
    }
    func (p *Person) String() string {
        return "Person - name is: " + p.Name + " - salary is: " + strconv.FormatFloat(p.Salary(), 'f', 2, 64)
    }
    func main() {
        bs := NewPerson("<NAME>", 2500.5)
        fmt.Println(bs)
        bs.SetSalary(4000.25)
        fmt.Println("Salary changed:")
        fmt.Println(bs)
    }
    ```












<file_sep>+++


date = "2015-07-12T14:59:31+11:00"
title = "Macbook Notes - Intel X64"
description = "Environment setup"
+++


### Prerequisites

* Mac OS 10.12+
* Intel X64 CPU


### Ownership issue

* If you have Homebrew or other software installed by someone else, you need to change ownership

```
sudo chown -R $(whoami) /usr/local/brew
sudo chown -R $(whoami) /usr/local/etc
sudo chown -R $(whoami) /usr/local/share
sudo chown -R $(whoami) /usr/local/lib

```

### Install Homebrew 

```bash
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

## Brew commands

brew update
brew upgrade 
brew list

```
{{% notice info %}}
The cask is not longer a brew command from 2021. The command has changed as below
{{% /notice %}}

```
# Before 2021
brew cask install XXXX

# After 2021
brew install --cask XXXX
```
### Install Zsh Prezto

- Clone Zsh Prezto

```sh
git clone --recursive https://github.com/sorin-ionescu/prezto.git "${ZDOTDIR:-$HOME}/.zprezto"

```

- Remove default `zshrc`

```sh
rm -f ~/.zshrc
```

- Initialize our Prezto configuration files.

```sh
setopt EXTENDED_GLOB
for rcfile in "${ZDOTDIR:-$HOME}"/.zprezto/runcoms/^README.md(.N); do
  ln -s "$rcfile" "${ZDOTDIR:-$HOME}/.${rcfile:t}"
done
```

- Setup Prezto Style

  - Open up ~/.zpreztorc and find where it says:
  - change “sorin” to “steeef.”

```sh
zstyle ':prezto:module:prompt' theme 'steeef'
```
  - Add Some Prezto Modules

```sh
'environment' \
  'terminal' \
  'editor' \
  'history' \
  'directory' \
  'spectrum' \
  'utility' \
  'completion' \
  'prompt' \
  'git' \
  'completion' \
  'syntax-highlighting' \
  'history-substring-search'
```


### Install NVM

```sh
brew install nvm
```


* Add following setting to file `.zshrc`

```sh
# NVM
export NVM_DIR="$HOME/.nvm"
  [ -s "/usr/local/opt/nvm/nvm.sh" ] && \. "/usr/local/opt/nvm/nvm.sh"  # This loads nvm
  [ -s "/usr/local/opt/nvm/etc/bash_completion.d/nvm" ] && \. "/usr/local/opt/nvm/etc/bash_completion.d/nvm"  # This loads nvm bash_completion
```

### Install python 3

```
brew install python@3.x
```


### Install MySql
{{% notice info %}}
Only the MySql 5.6 or 5.7 doesn't work well on Mac Big Sur or later. If you need old MySql, please consider to se docker to host mysql db.
{{% /notice %}}

```
brew install mysql
```



### Install JDK 


* Tap adoptopenjdk to brew

```sh
brew tap adoptopenjdk/openjdk

brew search openjdk
==> Formulae
openjdk                                     openjdk@11                                  openjdk@8                                   openj9                                      openvdb

==> Casks
adoptopenjdk-jre                     adoptopenjdk11-openj9                adoptopenjdk12-openj9-jre-large      adoptopenjdk14                       adoptopenjdk15-openj9                adoptopenjdk8
adoptopenjdk-openj9                  adoptopenjdk11-openj9-jre            adoptopenjdk12-openj9-large          adoptopenjdk14-jre                   adoptopenjdk15-openj9-jre            adoptopenjdk8-jre
adoptopenjdk-openj9-jre              adoptopenjdk11-openj9-jre-large      adoptopenjdk13                       adoptopenjdk14-openj9                adoptopenjdk15-openj9-jre-large      adoptopenjdk8-openj9
adoptopenjdk-openj9-jre-large        adoptopenjdk11-openj9-large          adoptopenjdk13-jre                   adoptopenjdk14-openj9-jre            adoptopenjdk15-openj9-large          adoptopenjdk8-openj9-jre
adoptopenjdk-openj9-large            adoptopenjdk12                       adoptopenjdk13-openj9                adoptopenjdk14-openj9-jre-large      adoptopenjdk16                       adoptopenjdk8-openj9-jre-large
adoptopenjdk10                       adoptopenjdk12-jre                   adoptopenjdk13-openj9-jre            adoptopenjdk14-openj9-large          adoptopenjdk16-jre                   adoptopenjdk8-openj9-large
adoptopenjdk11                       adoptopenjdk12-openj9                adoptopenjdk13-openj9-jre-large      adoptopenjdk15                       adoptopenjdk16-openj9                adoptopenjdk9
adoptopenjdk11-jre                   adoptopenjdk12-openj9-jre            adoptopenjdk13-openj9-large          adoptopenjdk15-jre                   adoptopenjdk16-openj9-jre
```

* Install multiple versions of JDK 

```
brew install --cask adoptopenjdk # Latest version is 16
brew install --cask adoptopenjdk8
brew install --cask adoptopenjdk11
```

* LTS version info

Java Version | First Release 		| Next Release 	| End of Availability 
------|--------|-----------|-----
Java 8 (LTS) 	 | Mar 2014 	|	jdk8u302 20th Jul 2021 |	At Least May 2026 
Java 11 (LTS)  |September 2018 | 	jdk-11.0.12 20th Jul 2021 	|At Least Oct 2024
Java 17 (LTS) 	|Sep 2021 	| jdk-17 14th Sep 2021 |	TBC


* Inspect all available JDKs

```
ls  /Library/Java/JavaVirtualMachines
adoptopenjdk-11.jdk adoptopenjdk-16.jdk adoptopenjdk-8.jdk

```

#### Switch Java version with alias
- Add following lines to file .zshrc

```sh
# Java
export JAVA_8_HOME=$(/usr/libexec/java_home -v1.8)
export JAVA_11_HOME=$(/usr/libexec/java_home -v11)
export JAVA_17_HOME=$(/usr/libexec/java_home -v16)

alias java8='export JAVA_HOME=$JAVA_8_HOME'
alias java11='export JAVA_HOME=$JAVA_11_HOME'
alias java17='export JAVA_HOME=$JAVA_16_HOME'

```

- Switch JDK

```sh
source ~/.zshrc

java8
java -version
java11 
java -version
```


#### Switch Java version with function

- Add following lines to file .zshrc

```sh
jdk() {
      if [[ ! -z $1 ]]; then
        version=$1
        unset JAVA_HOME;
        export JAVA_HOME=$(/usr/libexec/java_home -v"$version");
        java -version
      else
        echo Argument version is required. e.g. 1.8, 11, 16
        echo Example: jdk 1.8  or jdk 11
      fi
}
```

- Switch JDK

```sh
source ~/.zshrc

jdk
Argument version is required. e.g. 1.8, 11, 16
Example: jdk 1.8 or jdk 11

jdk 1.8

jdk 11

```


### Install Vim plugins


```
[[ ! -d ~/.vim/autoload ]] && mkdir -p ~/.vim/autoload
[[ ! -d ~/.vim/plugged ]] && mkdir -p ~/.vim/plugged

curl -fLo ~/.vim/autoload/plug.vim --create-dirs \
https://raw.githubusercontent.com/junegunn/vim-plug/master/plug.vim

cp ./.vimrc $HOME/

echo 'Launch vi and install plugins'
############################################################
# launch vim & install plugins by typing :PlugInstall
vi


```

### Install KubeCtl

```
brew unlink kubernetes-cli
brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/f25e36259eaa8bcf9b9add2c599aa6d8b15f437b/Formula/kubernetes-cli.rb

```

  
### Install Hugo

```
## Version: hugo_extended_0.54.0_macOS-64bit
tar -xvzf ~/Downloads/hugo_X.Y_osx-64bit.tgz
cp hugo_X.Y_osx-64bit.tgz
```


### Install AWS Cli 2.x

```
curl "https://awscli.amazonaws.com/AWSCLIV2-2.0.30.pkg" -o "AWSCLIV2.pkg"
sudo installer -pkg AWSCLIV2.pkg -target /
```


### Install Docker Desktop for mac

- Download & Install [Docker for Mac](https://docs.docker.com/desktop/mac/install/)

- Setup cli completion for zsh prezto

```
curl -fLo ~/.zprezto/modules/completion/external/src/_docker \
https://raw.githubusercontent.com/docker/cli/master/contrib/completion/zsh/_docker
```

- Add following line to zshrc to enable completion

```
autoload -Uz compinit; compinit
```


### Use Docker to launch databases


- Launch MySql 6.6 with docker


```
docker run -d --name mysql \
-p 3306:3306 \
-e MYSQL_ROOT_PASSWORD=<PASSWORD> \
mysql:5.6.51

```


- Access the MySql via bash

```
docker exec -it mysql bash

# After the bash into the docker container
mysql -u root -p
```

- Access the MySql via MySql client

```
mysql -h 0.0.0.0 \
      -u root  -p 
```


<file_sep>+++
title = "Golang"
description = "Golang Notes"
+++

{{%children style="card" description="true" sort="Weight" %}}<file_sep>+++

date = "2011-03-09T10:59:31+11:00"
title = "Windows cmd & hotkey - 1"
description="A note for everyone who wants to use Command and Hot Key as hacker "
weight=10
+++

*Do you want to make your friends amazed by your computer skill and praise you as genius? Or the hacker as watched in Sci-Fi movies? You don't need Mac, Linux or other operating systems, just Windows, you can show-off and look like hacker and master of Zeroes and Ones, even you have no any idea of it. Here are some tricks by which you can make your friends' jaw drop.*


## Start Windows command prompt as hacker

* Use use hotkeys to open `Run` feature in two keystrokes: `WinKey + R`
* Type `cmd` and press `Enter`

    ![Run](/img/windows-run-feature.png)

* Type `color A` to change the color of text to **Green**
* Change the title to **Hacker Tool**
* List the folders of current directory

```
C:\>User\<yourname>\color A
C:\>User\<yourname>\title Hacker Tool
C:\>User\<yourname>\cd \
C:\>tree 
```

## Use other command prompt 

* [cmder](http://cmder.net/) is an awesome product. I suggest you just choose mini version to download and install if you are not the heavy git user. There are so many built-in features you can play around.   
* [console2](https://sourceforge.net/projects/console/) is a very good as well. I used it for many years. I'm planning to migrate to `cmder`, but it will take me some time to do it, because I have some customized scripts need to run in `console2`. 

## Useful Windows hotkeys

I believe the common hotkeys you should know. e.g. `Ctrl + C, Ctrl + V, Ctrl + A`. Here the hotkeys I list below are some rarely-used but very useful hotkys. 

###  General hotkeys
 
* `Ctrl + Shift + Esc`  --  Open task manager
* `WinKey + R`  --  run dialog
* `Winkey + D`  --  toggle 'show desktop'
* `Winkey + L`  --  lock workstation
* `Winkey + E`  --  windows explorer
* `Ctrl + Shift + R`  --  clear page cache and refresh webpage on browser
* `Alt + (shift +) tab`  --  switch windows forwards (or backwards)
* `Alt + F4`  --  close the selected application
* `F2` , renames selected file. Also used with spreadsheet cells.
* `Ctrl and (+/-)`  --  zoom in or zoom out text on the editor tool
* Middle click a tab  --  close tab

###  Hotkeys for Windows 7 or higher version

* `WinKey + W`  --  search setting iterms
* `WinKey + Q`  --  search every iterms
* `WinKey + F`  --  search file iterms
* `WinKey + T`  --  use keyboard arrow keys to navigate dock
* `Winkey + X` - bring up laptop settings control panel
* `Ctrl +  N` - new tab 
* `Ctrl + Shift + N` - new Folder

###  Common short command lines for `Run` feature

* `cmd` -- start a Windows command prompt
* `calc` -- start the calculator application
* `notepad` -- start the notepad application 

###  Advanced short command lines for `Run` feature

* `mstsc` -- start the remote desktop application
* `regedit` -- start registry editor application
* `resmon`  --  awesome resource monitor - bandwidth etc (win7 or higher)
* `perfmon`  --  a pretty decent performance monitor (vista or higher)
* `services.msc`  --  windows service management
* `compmgmt.msc`  --  computer management including all other management 
* `eventvwr`--  windows event log viewer 
* `appwiz.cpl` -- windows programs and features management on control panel


## Most common and useful commands

This article won't list all commands and all usages of each command. Here I will just choose the commands and some usages of command which are useful for most people. Some advanced command will be explained in Part-2. 

Before you start typing any cmd, I want to share a common mistake for most beginners, including myself. We always forget to use help command before we Google a solution, when we hit some impediment or roadblock. Actually help command is the most common built-in feature within any software or tool. Learn how to use help command or find the help information is first important when we are going to learn anything new. 

###  help

* Start command prompt and type `help`. You will get a list of command which you can use, and short description of each command.
* Use help command to see and learn other commands
    ```
    C:\>help
    C:\>help cd 
    ```

###  start 

* Start another `cmd` window prompt. 

###  pwd 
* Type `pwd` to display current directory. All commands will use current directory as default path input if the path parameter is not specified. 

###  dir 
* Displays a list of files and subdirectories in a directory.

    * Type `dir /a:h/a:d` to display hidden subdirectories only
    * Type `dir /p/w` to display many items per screen within wide list format
    * Type `dir /o:-s` to display items sorted by size (biggest first)
    * Type `dir /o:s` to display items sorted by size (smallest first)
    * Type `dir /o:dn` to display items sorted by date/time  (oldest first) and name ( alphabetic)

###  cd / chdir 
* Displays the name of or changes the current directory
    * Type `cd` to display the name of directory 
    * Type `cd c:\windows` to change to `c:\Windows>` 
    * Type `cd /d D:` to change to d driver if you have d driver

###  tree  
* Use `tree /f/a <path>` to graphically display the folder structure of a drive or path. 

###  ren / rename
* Type `ren abc cba` to ren file name from "abc" to "cba" if there is file named abc under current directory. 
* Type `*.md *.txt` to ren all files under current directory with `md` extension to `txt` extension

###  md / mkdir
* Use `md a\b\c\d & tree a` to create all directories once and display result as follow

    ```
    <current-directory>\a
    |___b
        |___c
            |___d

    ```
    
###  copy 
* Use `touch test.txt & copy test.txt C:\User\<yourname>\` to create a test.txt file and copy the test.txt to C:\User\<yourname>\    

###  xcopy 
* Use `md a\b\c & touch a\test.txt & touch a\b\test.txt & xcopy /s /e /q a C:\User\<yourname>\a\` to folder a to `C:\User\<yourname>\`    
* Use `tree /f /a C:\User\<yourname>\a` to verify the result

###  move
* Type `move a b` to move folder `a` into folder `b`.


###  rd / rmdir
* Type `rd a` to remove a empty directory `a` 
* Type `rd /s a` to remove a directory `a` including all files and empty subdirectories within folder `a`.

###  del 
* __*IMPORTANT*__ : The item deleted by command `del` can not be restored from **Recyle Bin**. Please be careful before you use this command. 
* Type `del` to delete files or `del *.txt` to delete all files with `txt` extension



###  cls
* Type `cls` to clean the screen


<file_sep>+++

title = "Cassandra Practices"
description="Cassandra Introduction & Good practices ..."
draft = "false"

+++


_Apache Cassandra is a free open-source database system that is NoSQL based. Meaning Cassandra does not use the table model seen in MySQL, MSSQL or PostgreSQL, but instead uses a cluster model. It’s designed to handle large amounts of data and is highly scalable._




## Install Cassandra on Ubuntu

### Install Java(JRE) 8+

```bash
sudo add-apt-repository ppa:webupd8team/java
sudo apt update
sudo apt-get install oracle-java8-set-default
java -version
```

### Install Python 2.7+

```bash
sudo apt install python
python --version
```

## Install Cassandra

* First, we have to add Cassandra repository to source list by running following command. The `39x` is the version. Use 40x if Cassandra 4.0 is the newest version:

```bash
echo "deb http://www.apache.org/dist/cassandra/debian 39x main" | \
sudo tee -a /etc/apt/sources.list.d/cassandra.sources.list
```

* Next, run the cURL command to add the repository keys :

```bash
curl https://www.apache.org/dist/cassandra/KEYS | \
sudo apt-key add -
```

* We can now update the repositories and install Cassandra:

```bash 
sudo apt-get update
sudo apt install cassandra
# optional - It works on MacBook
sudo reboot
```

* Check Cassandra status

```bash
nodetool status
```

## Install Cassandra with Docker

### Create node n1

    docker run --name n1 -d tobert/cassandra -dc DC1 -rack RAC1
    docker ps

### Get IP of node n1

    IP=`docker inspect -f '{{ .NetworkSettings.IPAddress }}' n1`
    echo $IP


### Check status of n1

    docker exec -it n1  nodetool status

    # You will see status of Datacenter D1 below

    # Datacenter: DC1
    # ===============
    # Status=Up/Down
    # |/ State=Normal/Leaving/Joining/Moving
    # --  Address     Load       Tokens  Owns (effective)  Host ID   Rack
    UN  172.17.0.2  51.53 KB   256     100.0%    8965869d-cae8-41a6-bf19-ff69c2605c6c  RAC1

### Create node n2 on rack RAC2

    docker run --name=n2 -d tobert/cassandra -dc DC1 -rack RAC2 --seeds $IP

    docker exec -it n1  nodetool status

    # You will see status of Datacenter D1 below

    # Datacenter: DC1
    # ===============
    # Status=Up/Down
    # |/ State=Normal/Leaving/Joining/Moving
    # --  Address     Load       Tokens  Owns (effective)  Host ID   Rack
    # UN  172.17.0.3  72.01 KB   256     100.0%  cfa002b0-350c-41b8-9f86-eb8978a43b26  RAC2
    # UN  172.17.0.2  51.53 KB   256     100.0%  8965869d-cae8-41a6-bf19-ff69c2605c6c  RAC1


### Check node n2 configuration
 
    docker exec -it n2  cat /data/conf/cassandra.yaml | grep endpoint
    docker exec -it n2  cat /data/conf/cassandra-rackdc.properties | grep  -e "dc"  -e "rack"

### Create node n3 on Datacenter D2 and rack RAC1

    docker run --name=n3 -d tobert/cassandra -dc DC2 -rack RAC1 --seeds $IP
    docker exec -it n1  nodetool status

    # You will see status below. It may take a few seconds to get everything up and run.

    # Datacenter: DC1
    # ===============
    # Status=Up/Down
    # |/ State=Normal/Leaving/Joining/Moving
    # --  Address     Load    Tokens  Owns (effective)  Host ID  Rack
    # UN  172.17.0.3  134.03 KB  256     66.1%             cfa002b0-350c-41b8-9f86-eb8978a43b26  RAC2
    # UN  172.17.0.2  102.84 KB  256     64.5%             8965869d-cae8-41a6-bf19-ff69c2605c6c  RAC1

    # Datacenter: DC2
    # ===============
    # Status=Up/Down
    # |/ State=Normal/Leaving/Joining/Moving
    # --  Address   Load   Tokens  Owns (effective)  Host ID   Rack
    UN  172.17.0.4  14.38 KB   256     69.4%             0fad8335-763d-42fa-9934-3ed10c44eaa8  RAC1

### Setup replication strategy

    docker 
    create keyspace csdb with replication = {'class':'NetworkTopologyStrategy','DC1':2,'DC2':1}

### Check csdb status


    docker exec -it n1 nodetool describering csdb

    # Run nodetool status and note that the one node in DC2 owns all the data
    docker exec -it n1 nodetool status csdb  

    # Stop and remove all four docker containers:
    docker stop n1 n2 n3 n4; docker rm n1 n2 n3 n4


## Create single Datacenter with 3 nodes

### Create 3 nodes with local mounted directory

    docker run --name=n1 -v $PWD/ws/ps/cassdev/scripts:/scripts -d tobert/cassandra
    docker exec -it n1  nodetool status

    IP=`docker inspect -f '{{ .NetworkSettings.IPAddress }}' n1`
    echo $IP

    docker run --name=n2 -d tobert/cassandra --seeds $IP
    docker run --name=n3 -d tobert/cassandra --seeds $IP

    docker exec -it n1 /bin/bash
    cd scripts


### List scripts

    docker exec -it n1 /bin/bash
    cd /scrpts && ls


### Create keyspace with simple replication

    docker exec -it n1 cqlsh
    create keyspace csdb with replication = 
    {'class':'SimpleStrategy', 'replication_factor':1};

### Create table and insert data with script

    docker exec -it n1 cqlsh

    > desc keyspaces;
    > desc tables;
    > use ussdb;
    > source '/scripts/courses.cql'
    > source '/scripts/users.cql'


### CQL

    docker exec -it n1 cqlsh

    > use ussdb;
    > desc tables;
    > select * from users;
    > select * from courses;


### Troubleshoot

* Connection error

        Connection error: Could not connect to localhost:9160

    * Update the configuration - cassandra.yaml
    * Change the following IP address 
     
                - seeds: "xxx.xxx.xxx.xxx"
              listen_address: xxx.xxx.xxx.xxx
              broadcast_rpc_address: xxx.xxx.xxx.xxx

    * Restart the cassandra
    
            sudo systemctl restart cassandra




<file_sep>+++
title = "AWS Note - 5"
description = "Load Balanceing "
draft="true"
+++


## Load balancer

Elastic Load Balancing is regional service. It automatically distributes incoming application traffic across multiple targets, such as Amazon EC2 instances, containers, IP addresses, and Lambda functions. It can handle the varying load of your application traffic in a single Availability Zone or across multiple Availability Zones. 

### Application Load Balancer



#### Layer 7 load balancing (HTTP/HTTPS)

#### Server Name Indication (SNI)

#### IP addresses as targets

#### Lambda functions



### Network Load Balancer



<file_sep>+++
title="Goroutine & Channel - 1"
description="Goroutine & Channel - Part 1"
weight=6
+++


### Goroutine

* A goroutine is implemented as a function or method (this can also be an anonymous or lambda function) and called (invoked) with the keyword go. This starts the function running in parallel with the current computation but in the same address space and with its own stack.

* Go’s concurrency primitives provide the basis for a good concurrency program design: expressing program structure so as to represent independently executing actions; so Go’s emphasis is not in the 1 st place on parallelism: concurrent programs may or may not be parallel. Parallelism is the ability to make things run quickly by using multiple processors. But it turns out most often that a well designed concurrent program also has excellent performing parallel capabilities.

#### Basic Goroutine

* Sample 1: The code below will not print out anything, because the main exits before the "hello" goroutine kicks off. Actually the main is another goroutine which always runs first. 

```go
func main() {
    go func() {
        println("Hello")
    }()
}
```

* Sample 2: Add timer to set main goroutine sleep a while, and let the "hello"
goroutine to start.

```go
func main() {
    go func() {
        println("Hello") // Hello
    }()
    time.Sleep(3000)
}
```

* Sample 3: The code below will print "hello" 10 times and then print "world" 10 times. 

```go
func main() {
    go func() {
        for i := 0; i < 10; i++ {
            println("Hello")
            time.Sleep(2000)
        }
    }()
    go func() {
        for i := 0; i < 10; i++ {
            println("World")
        }
    }()
    time.Sleep(9999)
}
```

* Sample 4: The code below will print "hello" and "world" in different order.

```go
func main() {
    runtime.GOMAXPROCS(4)
    go func() {
        for i := 0; i < 10; i++ {
            println("Hello")
            time.Sleep(500)
        }

    }()
    go func() {
        for i := 0; i < 10; i++ {
            println("World")
            time.Sleep(500)
        }
    }()
    time.Sleep(999999)
}
```

* Sample 5: File watcher

```go

const watchedPath = "./source"

func main() {
    for {
        d, _ := os.Open(watchedPath)
        files, _ := d.Readdir(-1)
        for _, fi := range files {
            filePath := watchedPath + "/" + fi.Name()
            f, _ := os.Open(filePath)
            data, _ := ioutil.ReadAll(f)
            f.Close()
            os.Remove(filePath)
            go func(data string) {
                reader := csv.NewReader(strings.NewReader(data))
                records, _ := reader.ReadAll()
                for _, r := range records {
                    invoice := new(Invoice)
                    invoice.Number = r[0]
                    invoice.Amount, _ = strconv.ParseFloat(r[1], 64)
                    invoice.PurchaseOrderNumber, _ = strconv.Atoi(r[2])
                    unixTime, _ := strconv.ParseInt(r[3], 10, 64)
                    invoice.InvoiceDate = time.Unix(unixTime, 0)
                    
                    fmt.Printf("Received Invoice '%v' for $%.2f and submitted for processing\n", invoice.Number, invoice.Amount)
                }
            }(string(data))
        }
        d.Close()
        time.Sleep(100 * time.Millisecond)
    }
}

type Invoice struct {
    Number string
    Amount float64
    PurchaseOrderNumber int
    InvoiceDate time.Time
}
```


#### Using GOMAXPROCS

* When GOMAXPROCS is greater than 1, they run on a thread pool with that many threads.With the gccgo compiler GOMAXPROCS is effectively equal to the number of running goroutines.

* Observations from experiments: on a 1 CPU laptop performance improved when GOMAXPROCS was increased to 9. On a 32 core machine, the best performance was reached with GOMAXPROCS=8, a higher number didn’t increase performance in that benchmark.

### Channels

#### Channels for communication between goroutines

* Go has a special type, the channel, which is a like a conduit (pipe) through which you can send typed values and which takes care of communication between goroutines, avoiding all the pitfalls of shared memory; the very act of communication through a channel guarantees synchronization.


* Data are passed around on channels: only one goroutine has access to a data item at any given time: so data races cannot occur, by design. The ownership of the data (that is the ability to read and write it) is passed around.

* A channel is in fact a typed message queue: data can be transmitted through it. It is a First In First Out (FIFO) structure and so they preserve the order of the items that are sent into them (for those who are familiar with it, a channel can be compared to a two-way pipe in Unix shells).

* A channel is also a reference type, so we have to use the make() function to allocate memory for it.

**NOTE: Don’t use print statements to indicate the order of sending to and receiving from a channel: this could be out of order with what actually happens due to the time lag between the print statement and the actual channel sending and receiving.**


#### Blocking of channels

* A send operation on a channel (and the goroutine or function that contains it) blocks until a receiver is available for the same channel

* A receive operation for a channel blocks (and the goroutine or function that contains it) until a sender is available for the same channel
 


##### Goroutine sync through one or more channels

* Blocking - deadlock

* Sample 1 :  deadlock because of lack of sender

```go
func main() {
    ch := make(chan string)
    fmt.Println(<-ch)
}
//fatal error: all goroutines are asleep - deadlock!
```

* Sample 2: deadlock because of lack of receiver

```go
func main() {
    ch := make(chan string, 1)
    ch <- "Hello"
    
}
```

* Sample 3 

```go
func main() {
    ch := make(chan string, 1)
    ch <- "Hello"
    fmt.Println(<-ch) // Hello
}
```



##### Async channels - channel with buffer

* Channel with buffer

```go
buf := 100
ch1 := make(chan string, buf)
```


* Sample 4: deadlock again because the 2nd sender can not send message

```go
func main() {
    ch := make(chan string, 1)
    ch <- "Hello"
    ch <- "Hello"
    fmt.Println(<-ch)
    fmt.Println(<-ch)
}
```

* Sample 5 

```go
func main() {
    ch := make(chan string, 2)
    ch <- "Hello"
    ch <- "Hello"
    fmt.Println(<-ch) //Hello
    fmt.Println(<-ch) // Hello
}
```

##### Closing channel



* Sample 6: Closing channel will not impact the receiver to get the message

```go
func main() {
    ch := make(chan string, 1)
    ch <- "Hello"
    ch <- "Hello"
    fmt.Println(<-ch)
    fmt.Println(<-ch)
}
```

* Sample 7: Sender cannot send message to closing channel 

```go
func main() {
    ch := make(chan string, 2)
    ch <- "Hello"
    ch <- "Hello"
    fmt.Println(<-ch) //Hello
    fmt.Println(<-ch) // Hello
    ch <- "Hello" // panic: send on closed channel
}
```


#### Semaphore pattern

* The goroutine compute signals its completion by putting a value on the channel ch, the main routine waits on <-ch until this value gets through.

```go
func compute(ch chan int) {
    ch <- someComputation()
    // when it completes, signal on the channel.
}

func main() {
    ch := make(chan int) // allocate a channel.
    go compute(ch)       // start something in a goroutine
    doSomethingElseForAWhile()
    result := <-ch
}
```

##### Implement a semaphore with a buffered channel

* There is no semaphore implementation in Go’s sync package, but they can be emulated easily using a buffered channel:
    
    *  the buffered channel is the number of resources we wish to synchronize
    * the length (number of elements currently stored) of the channel is the number of resources currently being used.
    * the capacity minus the length of the channel is the number of free resources (the integer value of traditional semaphores)


* Sample 8: semaphore pattern

```go
type Empty interface {}
var empty Empty
//  do sth ...
data := make([]float64, N)
res := make([]float64, N)
sem := make(chan Empty, N) // semaphore
// do sth  ...
for i, xi := range data {
    go func (i int, xi float64) {
        res[i] = doSomething(i,xi)
        sem <- empty
    } (i, xi)
}
// wait for goroutines to finish
for i := 0; i < N; i++ { <-sem }

```

* Semaphore operations sample pattern

```go
// acquire n resources
func (s semaphore) P(n int) {
    e := new(Empty)
    for i := 0; i < n; i++ {
        s <- e
    }
}
// release n resources
func (s semaphore) V(n int) {
    for i := 0; i < n; i++ {
        <-s
    }
}
```

* Semaphore for a mutex:

```go
/* mutexes */
func (s semaphore) Lock() {
    s.P(1)
}
func (s semaphore) Unlock() {
    s.V(1)
}
/* signal-wait */
func (s semaphore) Wait(n int) {
    s.P(n)
}
func (s semaphore) Signal() {
    s.V(1)
}
```



#### Channel Factory pattern

* Another pattern common in this style of programming goes as follows: instead of passing a channel as a parameter to a goroutine, let the function make the channel and return it (so it plays the role of a factory); inside the function a lambda function is called as a goroutine.


```go
func main() {
    stream := pump()
    go suck(stream)
    // the above 2 lines can be shortened to: go suck( pump() )
    time.Sleep(1e9)
}
func pump() chan int {
    ch := make(chan int)
    go func() {
        for i := 0; ; i++ {
            ch <- i
        }
    }()
    return ch
}
func suck(ch chan int) {
    for {
        fmt.Println(<-ch)
    }
}
```

##### For—range applied to channels

* The range clause on for loops accepts a channel ch as an operand, in which case the for loops over the values received from the channel. 

* Obviously another goroutine must be writing to ch (otherwise the execution blocks in the for-loop) and must close ch when it is done writing. 

```go

func main() {
    suck(pump())
    time.Sleep(1e9)
}
func pump() chan int {
    ch := make(chan int)
    go func() {
        for i := 0; ; i++ {
            ch <- i
        }
    }()
    return ch
}
func suck(ch chan int) {
    go func() {
        for v := range ch {
            fmt.Println(v)
        }
    }()
}
```

* 

##### Iterator pattern

* Another common case where we have to populate a channel with the items of a container type which contains an index-addressable field items . For this we can define a method which returns a read-only channel.

* Inside the goroutine, a for-loop iterates over the elements in the container c (for tree or graph algorithms, this simple for-loop could be replaced with a depth-first search)


    ```go
    func (c *container) Iter() <-chan items {
        ch := make(chan item)
        go func() {
            for i := 0; i < c.Len(); i++ {
                // or use a for-range loop
                ch <- c.items[i]
            }
        }()
        return ch
    }

    // The code which calls this method can then iterate over the container
    for x := range container.Iter() { ... }

    ```


#### Producer Consumer pattern 

* A Produce() function which delivers the values needed by a Consume function. Both could be run as a separate goroutine, Produce putting the values on a channel which is read by Consume. 

    ```go
    package main
    /* producer-consumer problem in Go */

    import ("fmt")

    var done = make(chan bool)
    var msgs = make(chan int)

    func produce () {
        for i := 0; i < 10; i++ {
            msgs <- i
        }
        done <- true
    }

    func consume () {
        for {
            msg := <-msgs
            fmt.Println(msg)
        }
    }

    func main () {
        go produce()
        go consume()
        <- done
    }
    ```
















<file_sep>+++
title = "Thread & Task"
description = "Introduction of Thread & Task "
+++


## C# Thread & Task

> From Net 4.0, .Net applicatoin introduced Parallel Framework Extensions (PFx), along the way it delivered an entirely new model for async processing in
.NET. In .NET 4.0 the thread pool queue was redesigned with the new requirements of PFx in mind. Instead of using a simple linked list, the queue was built with arrays of work items with the arrays connected into a linked list.

> With the release of .NET 4.0, Microsoft introduced yet another API for building asynchronous applications: the Task Parallel Library (TPL).
 

### Old way of C# thread

* Early days of .NET applications you will see many following sample codes.

## Lock keyword

```cs
    lock(stateGuard)
    {
        cash += amount;
        receivables -= amount;
    }
```

## Abort 

`Thread.Abort` should be avoid at all costs. Its behavior is much safer and predictable since .NET 2.0, but there are still some pretty serious pitfalls with it. The other ways to stop thread.
* Use `Thread.Interrupt`
* Use a stopping flag
* Use `WaitHandle` events

### New way of C# Async & Thread

## Monitor 

*  Monitor is no different from lock but the monitor class provides more control over the synchronization of various threads trying to access the same lock of code.

* Lock and Monitor sample

```cs
class Program  
{  
    static readonly object _object = new object();  

    public static void PrintNumbers()  
    {  
        Monitor.Enter(_object);  
        try  
        {  
            for (int i = 0; i < 5; i++)  
            {  
                Thread.Sleep(100);  
                Console.Write(i + ",");  
            }  
            Console.WriteLine();  
        }  
        finally  
        {  
            Monitor.Exit(_object);  
        }  
    }  

    static void TestLock()  
    {            
        lock (_object)  
        {  
            Thread.Sleep(100);  
            Console.WriteLine(Environment.TickCount);  
        }  
    }  

    static void Main(string[] args)      
    {            
        Thread[] Threads = new Thread[3];  
        for (int i = 0; i < 3; i++)  
        {  
            Threads[i] = new Thread(new ThreadStart(PrintNumbers));  
            Threads[i].Name = "Child " + i;  
        }  
        foreach (Thread t in Threads)  
            t.Start();  

        Console.ReadLine();  
    }  
}  
```

* Once you have a lock on a code region, you can use the Monitor.Wait, Monitor.Pulse, and Monitor.PulseAll methods.

* Lock and monitor are basically used for the same purpose in multithreading, the difference is that only when we want more control over synchronization with multiple threads running for a specific section of code.


### Monitor Semaphore


```cs
public class MonitorSemaphore
{
    private int currentCount;
    private readonly int maxCount;
    private readonly object guard = new object();
    public MonitorSemaphore(int initialCount, int maxCount)
    {
        this.currentCount = initialCount;
        this.maxCount = maxCount;
    }

    public void Enter()
    {
        lock (guard)
        {
            while (currentCount == maxCount)
            {
                Monitor.Wait(guard);
            }
            currentCount++;
        }
    }

    public void Exit()
    {
        lock (guard)
        {
            currentCount--;
            Monitor.Pulse(guard);
        }
    }
    public int CurrentCount{get { return currentCount; }}
}
```

<file_sep>+++

date = "2020-02-07T16:56:21+11:00"
title = "Angular 9 CRM Starter Project"
description="Ng Crm is reusable CRM project for real-world business based on Angular 9"
+++




###  Key Takeaways
  
* Angular 9 is here now. 
* Angular Material 9 is ready as well.
* Reusable starter project

### Summary

**Ng-Crm** > A reusable CRM starter project for real-world business based on Angular 9, Angular-Material 9.x.

This project was generated with [Angular CLI](https://github.com/angular/angular-cli) version 9.x. The goal of this project is to create reusable project for real-world business. To achieve this target, we need a solution which includes simple authentication process, restful API feature with token support and simple but elegant UI design.

#### Features

* This project is built on the top of AngularClass/Angular-Starter.
* The UI part of this project is comprehensively built on Angular Material.
* This project includes ng-charts.
* ~~To simulate real-world business, this starter project chooses Json-Server as fake Restful API. (You can simple replace it with your own API).~~
* ~~Fake API is just readonly fake service.~~



#### Live Demo
[Demo App](https://angular-app-demo.harryho.org): The demo is just a proof of concept. It doesn't have back-end API and all features of master branch.


### Build & Setup

```bash
# Clone project
git clone https://github.com/harryho/ng-crm.git


# Install Angular CLI
npm install -g @angular/cli

# prepare Json-Server as fake Restful API
cd ng-crm


# Install the packages with npm
npm install


# Start the app with npm
npm start
# Or use ng
ng serve 

# Test with npm
npm run test
# Or use ng
ng test


# build for production 
npm run build --prod=true

# run as production
install -g serve
serve dist

```

### Docker

```
## Run / Test release without building new image
npm run build

# Launch nginx image to test latest release
docker pull nginx:alpine
docker run -p 8080:80 -v \
    <your_aboslute_path>/dist:/usr/share/nginx/html nginx:alpine


# Build release image
docker build . -t  nc-prd:2.0

# Launch the development image in the backgroud
docker run -d --publish 8080:80  --name nc2 nc-prd:2.0

# Check the log
docker logs nc2  -f
```

### Welcome to fork or clone!

For detailed explanation on how things work, checkout following links please.

* [angular](https://angular.io/)
* [angular-material](https://material.angular.io/)
* [ng-charts](https://github.com/valor-software/ng2-charts)
* [rxjs](https://rxjs.dev/api)



### Structure of ng-crm

``` ini
├── angular.json            <-// configuration of dev or prod 
├── config
│   └── nginx.conf         <-// nginx config for deployment 
├── src
│   ├── app
|   |   +---_gurad               <-// auth guard for authentication
|   |   +---_models              <-// common models for whole app
|   |   +---_services            <-// common services for whole app
|   |   +---about                <-// about component   
|   |   +---customer             <-// customer component
|   |   +---dashboard            <-// dashboard component  
|   |   +---notfoundpage         <-// notfoundpage component  
|   |   +---login                <-// login component  
|   |   +---order                <-// customer component 
|   |   +---root                 <-// root component 
|   |   +---shared               <-// common component for whole app
│   │   ├── app-routing.module.ts
│   │   ├── app.component.html
│   │   ├── app.component.scss
│   │   ├── app.component.spec.ts
│   │   ├── app.component.ts
│   │   ├── app.module.ts
│   ├── assets               <-//   images  and css from third parties
│   │   └── img
|── Dockerfile               <-// project comes with docker
├── tsconfig.app.json
├── tsconfig.json
├── tsconfig.spec.json
└── tslint.json
|__ ....
```


### Screenshots

> ![Screenshot1](/img/ng-crm-screenshot-1.jpg)

> ![Screenshot2](/img/ng-crm-screenshot-2.jpg)

> ![Screenshot3](/img/ng-crm-screenshot-3.jpg)

> ![Screenshot4](/img/ng-crm-screenshot-4.jpg)



### Browse [Repository](https://github.com/harryho/ng-crm.git)


### __Alternatives__

There are two similar projects respectively built on the Vue.js and React. If you have interests in those technical stacks. You can find and clone those projects below.

* [Vue2 Crm](/projects/vue2-crm).
* [React Redux Crm](/projects/react-crm).
<file_sep>+++
title = "F# Lazy"
description = "F# Computation - Lazy expressions"
weight = 14
+++

## Lazy Expressions


Lazy expressions are expressions that are not evaluated immediately, but are instead evaluated when the result is needed. This can help to improve the performance of your code.

`let identifier = lazy ( expression )`


Lazy expressions enable you to improve performance by restricting the execution of an expression to only those situations in which a result is needed.

To force the expressions to be performed, you call the method Force. Force causes the execution to be performed only one time. Subsequent calls to Force return the same result, but do not execute any code.


```fsharp
let x = 10
let result = lazy (x + 10)
printfn "%d" (result.Force())
```

Lazy evaluation, but not the Lazy type, is also used for sequences.<file_sep>+++

date = "2017-08-03T16:56:21+11:00"
title = "FlatApi - Restful API for python dev"

description = "FlatApi is a zero coding and zero configuration restful API server inspired by Json-Server and Eve"
+++

## Summary

FlatApi is a **zero coding** and **zero configuration** restful API server inspired by Json-Server_ and Eve_. It is designed to be used as fake restful api for development, especially for people want to use Python stack. Setup process is **less than 10 seconds**. 


## FlatApi

- **Zero coding and configuration to setup Restful API** FlatApi is designed to use without coding and configuration by default. You just need one config to setup all endpoints you need, then you can use it immediately. 

- **Flask based web server** FlatApi is built on the top of _Flask

- **Json flat file database** FlatApi uses FlatApi_ to manage the Json flat file database. FlatApi is a document oriented database. 

- **Caching memory storage availble** FlatApi supports caching momery storage after version 4.0.0. 


### Install Package


```bash
$ pip uninstall flatapi
$ pip install --no-cache-dir flatapi
```

### Quick Start


- Launch FlatApi without configuration

```bash
## Start the FlatApi - Sample 1 
$ python3 /<path_to_package>/flatapi -S MEMORY -G NO
## Start the FlatApi - Sample 2
$ python3 /<path_to_package>/flatapi --storage MEMORY -cfgfile NO
## Start the FlatApi with prefix - Sample 3
$ python3 /<path_to_package>/flatapi --storage memory -cfgfile no -X api



\(^_^)/ Hi

Loading  is done.

There is no config file found. Flat Api uses internal configuration.

Resource :
/<string:doc> -- The doc is the collection name
                    you want to post or put the object.
/<string:doc>/<int:id> --The id is the unique id for query or delete.

Database: Memory

* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
```

- Test api via postman 

    It would be a much more handy and easy way to play around with the API immediately.

```bash
GET /posts       --> Get all posts
POST /posts      --> Add new post
PUT /posts/1     --> Update existing post which id is 1
DELETE /posts/1  --> Delete a post which id is 1
DELETE /posts    --> Delete all posts
```

- Test api via curl 

```bash
## Add a new post
$ curl -d "{\"text\":\"post 1\",\"author\":\"harry\"}" -H "Content-Type: application/json" -X POST http://localhost:5000/posts
{"author": "harry", "text": "post 1", "id": 1}

## Get post by Id
$ curl -X GET http://localhost:5000/posts/1
{"author": "harry", "text": "post 1", "id": 1}

## Get all posts
$ curl -X GET http://localhost:5000/posts
[{"author": "harry", "text": "post 1", "id": 1}]

## Update  the post
$ curl -d "{\"text\":\"post updated\",\"author\":\"harry\"}" -H "Content-Type: application/json" -X PUT http://localhost:5000/posts/1
[{"author": "harry", "text": "post updated", "id": 1}]

## Delete 
$ curl -X DELETE http://localhost:5000/posts 
```
  
### Browse [Repository](https://github.com/harryho/flat-api.git)<file_sep>+++
title = "AWS: VPC - 5"
description = "VPC Peering, Direct Connect, Transit Gateway"
weight=6
+++


## VPC Part 5


### Direct Connect

AWS Direct Connect is a cloud service solution that makes it easy to establish a dedicated network connection from your premises to AWS. Using AWS Direct Connect, you can establish private connectivity between AWS and your datacenter, office, or colocation environment, which in many cases can reduce your network costs, increase bandwidth throughput, and provide a more consistent network experience than Internet-based connections.

Using industry standard 802.1q VLANs, this dedicated connection can be partitioned into multiple virtual interfaces. This allows you to use the same connection to access public resources such as objects stored in Amazon S3 using public IP address space, and private resources such as Amazon EC2 instances running within an Amazon Virtual Private Cloud (VPC) using private IP space, while maintaining network separation between the public and private environments. Virtual interfaces can be reconfigured at any time to meet your changing needs.


### Transit Gateway

AWS Transit Gateway is a service that enables customers to connect their Amazon Virtual Private Clouds (VPCs) and their on-premises networks to a single gateway.

With AWS Transit Gateway, you only have to create and manage a single connection from the central gateway in to each Amazon VPC, on-premises data center, or remote office across your network. Transit Gateway acts as a hub that controls how traffic is routed among all the connected networks which act like spokes. This hub and spoke model significantly simplifies management and reduces operational costs because each network only has to connect to the Transit Gateway and not to every other network. Any new VPC is simply connected to the Transit Gateway and is then automatically available to every other network that is connected to the Transit Gateway. This ease of connectivity makes it easy to scale your network as you grow.


<file_sep>+++
title = "Generic Type & Trait "
description="Rustlang Introduction: Generic Type, Trait and Lifetime "
weight = 4
+++


### Generic Type

* Generics are abstract stand-ins for concrete types or other properties. When we’re writing code, we can express the behavior of generics or how they relate to other generics without knowing what will be in their place when compiling and running the code.


#### Removing Duplication by Extracting a Function

* steps we took to change the duplication code :

    * Identify duplicate code.
    
    * Extract the duplicate code into the body of the function and specify the inputs and return values of that code in the function signature.
    
    * Update the two instances of duplicated code to call the function instead.



* Defining a function makes our code more flexible and provides more functionality to callers of our function while preventing code duplication.


#### Definition with generic type

* Define structs to use a generic type parameter in one or more fields using the <> syntax. 

* A sample below which won't compile

    ```rs
    struct Point<T> {
        x: T,
        y: T,
    }

    fn main() {
        let wont_work = Point { x: 5, y: 4.0 };
        //                           ^^^ expected integral variable, found
        //                   floating-point variable
        //                   note: expected type `{integer}`
    }
    ```

* Above sample can be refactored as below to make it work

    ```rs
    struct Point<Y,U>{
        x: T,
        y: U,
    }
    ```

* define enums to hold generic data types in their variants. 

    ```rs
    enum Option<T> {
        Some(T),
        None,
    }

    enum Result<T, E> {
        Ok(T),
        Err(E),
    }
    ```

* implement methods on structs and enums, and use generic types in their definitions

    ```rs
    struct Point<T> {
        x: T,
        y: T,
    }

    impl<T> Point<T> {
        fn x(&self) -> &T {
            &self.x
        }
    }

    fn main() {
        let p = Point { x: 5, y: 10 };

        println!("p.x = {}", p.x());
    }
    ```

*  implement methods with concrete type f32, meaning we don’t declare any types after impl.

    ```rs
    impl Point<f32> {
        fn distance_from_origin(&self) -> f32 {
            (self.x.powi(2) + self.y.powi(2)).sqrt()
        }
    }
    ```


* Generic type parameters in a struct definition aren’t always the same as those you use in that struct’s method signatures. 

* Sample below the method mixup on the Point<T, U> struct from Listing 10-8. The method takes another Point as a parameter, which might have different types from the self Point we’re calling mixup on. The method creates a new Point instance with the x value from the self Point (of type T) and the y value from the passed-in Point (of type W).

    ```rs
    struct Point<T, U> {
        x: T,
        y: U,
    }

    impl<T, U> Point<T, U> {
        fn mixup<V, W>(self, other: Point<V, W>) -> Point<T, W> {
            Point {
                x: self.x,
                y: other.y,
            }
        }
    }

    fn main() {
        let p1 = Point { x: 5, y: 10.4 };
        let p2 = Point { x: "Hello", y: 'c'};

        let p3 = p1.mixup(p2);

        println!("p3.x = {}, p3.y = {}", p3.x, p3.y); // p3.x = 5, p3.y = c
    }
    ```

* The purpose of above example is to demonstrate a situation in which some generic parameters are declared with impl and some are declared with the method definition. Here, the generic parameters T and U are declared after impl, because they go with the struct definition. The generic parameters V and W are declared after fn mixup, because they’re only relevant to the method.

#### Performance of Code Using Generics

*  The good news is that Rust implements generics in such a way that your code doesn’t run any slower using generic types than it would with concrete types.

* Rust accomplishes this by performing monomorphization of the code that is using generics at compile time. Monomorphization is the process of turning generic code into specific code by filling in the concrete types that are used when compiled.

* Sample of Rust compile the generic type

    ```rs
    let integer = Some(5);
    let float = Some(5.0);

    // Above is generic type 
    // ------------------- 

    // Rust will create specific definition as following
    enum Option_i32 {
        Some(i32),
        None,
    }

    enum Option_f64 {
        Some(f64),
        None,
    }

    fn main() {
        let integer = Option_i32::Some(5);
        let float = Option_f64::Some(5.0);
    }
    ```

* Rust compiles this code, it performs monomorphization. During that process, the compiler reads the values that have been used in Option<T> instances and identifies two kinds of Option<T>: one is i32 and the other is f64. As such, it expands the generic definition of Option<T> into Option_i32 and Option_f64, thereby replacing the generic definition with the specific ones.

* Because Rust compiles generic code into code that specifies the type in each instance, there is no runtime cost for using generics. When the code runs, it performs just as it would if we had duplicated each definition by hand. The process of monomorphization makes Rust’s generics extremely efficient at runtime.


### Traits: Defining Shared Behavior

* A trait tells the Rust compiler about functionality a particular type has and can share with other types. We can use trait bounds to specify that a generic can be any type that has certain behavior.

#### Defining a trait

* A type’s behavior consists of the methods we can call on that type. Different types share the same behavior if we can call the same methods on all of those types. Trait definitions are a way to group method signatures together to define a set of behaviors necessary to accomplish some purpose.


* Sample of trait

    ```rs
    pub trait Summary {
        fn summarize(&self) -> String;
    }
    ```

#### Implementing a trait

* Implementing a trait on a type is similar to implementing regular methods. The difference is that after impl, we put the trait name that we want to implement, then use the for keyword, and then specify the name of the type we want to implement the trait for. 


    ```rs
    pub struct NewsArticle {
        pub headline: String,
        pub location: String,
        pub author: String,
        pub content: String,
    }

    impl Summary for NewsArticle {
        fn summarize(&self) -> String {
            format!("{}, by {} ({})", self.headline, self.author, self.location)
        }
    }

    pub struct Tweet {
        pub username: String,
        pub content: String,
        pub reply: bool,
        pub retweet: bool,
    }

    impl Summary for Tweet {
        fn summarize(&self) -> String {
            format!("{}: {}", self.username, self.content)
        }
    }
    ```

    * Test the implementation

    ```rs
    let tweet = Tweet {
        username: String::from("horse_ebooks"),
        content: String::from("of course, as you probably already know, people"),
        reply: false,
        retweet: false,
    };

    println!("1 new tweet: {}", tweet.summarize());
    ```

#### Default implementation

* Sample 

    ```rs
    pub trait Summary {
        fn summarize(&self) -> String {
            String::from("(Read more...)")
        }
    }
    ```

* Default implementations can call other methods in the same trait, even if those other methods don’t have a default implementation. In this way, a trait can provide a lot of useful functionality and only require implementors to specify a small part of it. 

    ```rs
    pub trait Summary {
        fn summarize_author(&self) -> String;

        fn summarize(&self) -> String {
            format!("(Read more from {}...)", self.summarize_author())
        }
    }

    impl Summary for Tweet {
        fn summarize_author(&self) -> String {
            format!("@{}", self.username)
        }
    }

    let tweet = Tweet {
        username: String::from("horse_ebooks"),
        content: String::from("of course, as you probably already know, people"),
        reply: false,
        retweet: false,
    };

    println!("1 new tweet: {}", tweet.summarize());
    ```

#### Traits as Parameters

* use traits to define functions that accept many different types.

* Sample: Instead of a concrete type for the item parameter, we specify the impl keyword and the trait name. 

    ```rs
    pub fn notify(item: impl Summary) {
        println!("Breaking news! {}", item.summarize());
    }
    ```

* Anotehr sample

    ```rs
    pub fn notify<T: Summary>(item1: T, item2: T) {}
    ```

#### Clearer Trait Bounds with where Clauses

* Using too many trait bounds has its downsides. Each generic has its own trait bounds, so functions with multiple generic type parameters can contain lots of trait bound information between the function’s name and its parameter list, making the function signature hard to read. 

* Sample with where

    ```rs
    fn some_function<T, U>(t: T, u: U) -> i32
        where T: Display + Clone,
            U: Clone + Debug {}
    ```

#### Returning Types that Implement Traits

*  use the impl Trait syntax in the return position to return a value of some type that implements a trait

* However, you can only use impl Trait if you’re returning a single type

```rs
fn returns_summarizable() -> impl Summary {
    Tweet {
        username: String::from("horse_ebooks"),
        content: String::from("of course, as you probably already know, people"),
        reply: false,
        retweet: false,
    }
}
```

#### Fixing the largest Function with Trait Bounds

* Without the PartialOrd trait, the largest will throw error "an implementation of `std::cmp::PartialOrd` might be missing for `T`"

* Without the Copy trait, the largest function will throw compilation error as well

    ```rs
    fn largest<T: PartialOrd + Copy>(list: &[T]) -> T {
        let mut largest = list[0]; // Without copy -- error: "cannot move out of here"  

        for &item in list.iter() {
            if item > largest { // // Without copy -- error: "cannot move out of borrowed content"
                largest = item;
            }
        }

        largest
    }

    fn main() {
        let number_list = vec![34, 50, 25, 100, 65];

        let result = largest(&number_list);
        println!("The largest number is {}", result);

        let char_list = vec!['y', 'm', 'a', 'q'];

        let result = largest(&char_list);
        println!("The largest char is {}", result);
    }
    ```



#### Using Trait Bounds to Conditionally Implement Methods

* By using a trait bound with an impl block that uses generic type parameters, we can implement methods conditionally for types that implement the specified traits. 


    ```rs
    #![allow(unused_variables)]
    fn main() {
    use std::fmt::Display;

    struct Pair<T> {
        x: T,
        y: T,
    }

    impl<T> Pair<T> {
        fn new(x: T, y: T) -> Self {
            Self {
                x,
                y,
            }
        }
    }

    impl<T: Display + PartialOrd> Pair<T> {
        fn cmp_display(&self) {
            if self.x >= self.y {
                println!("The largest member is x = {}", self.x);
            } else {
                println!("The largest member is y = {}", self.y);
            }
        }
    }
    ```

### Lifetimes

#### Preventing Dangling References with Lifetimes

* Rust requires us to annotate the relationships using generic lifetime parameters to ensure the actual references used at runtime will definitely be valid.

* The main aim of lifetimes is to prevent dangling references, which cause a program to reference data other than the data it’s intended to reference. 


    ```rs
    {
        let r;                // ---------+-- 'a
                              //          |
        {                     //          |
            let x = 5;        // -+-- 'b  |
            r = &x;           //  |       |
        }                     // -+       |
                              //          |
        println!("r: {}", r); //          |
    }                         // ---------+
    ```

#### Generic Lifetimes in Functions

*



#### Lifetime Annotation Syntax

```rs
&i32        // a reference
&'a i32     // a reference with an explicit lifetime
&'a mut i32 // a mutable reference with an explicit lifetime
```


#### Lifetime Annotations in Function Signatures


```rs
#![allow(unused_variables)]
fn main() {
    let string1 = String::from("abcd");
    let string2 = "xyz";

    let result = longest(string1.as_str(), string2);
    println!("The longest string is {}", result);
}


fn longest<'a>(x: &'a str, y: &'a str) -> &'a str {
    if x.len() > y.len() {
        x
    } else {
        y
    }
}


```



#### Lifetime Elision

```rs

fn first_word(s: &str) -> &str {
    let bytes = s.as_bytes();

    for (i, &item) in bytes.iter().enumerate() {
        if item == b' ' {
            return &s[0..i];
        }
    }

    &s[..]
}
```

* The patterns programmed into Rust’s analysis of references are called the lifetime elision rules. These aren’t rules for programmers to follow; they’re a set of particular cases that the compiler will consider, and if your code fits these cases, you don’t need to write the lifetimes explicitly.

* The elision rules don’t provide full inference. If Rust deterministically applies the rules but there is still ambiguity as to what lifetimes the references have, the compiler won’t guess what the lifetime of the remaining references should be. 

* Lifetimes on function or method parameters are called input lifetimes, and lifetimes on return values are called output lifetimes.


#### Lifetime Elision Rules

* The compiler uses three rules to figure out what lifetimes references have when there aren’t explicit annotations. The first rule applies to input lifetimes, and the second and third rules apply to output lifetimes. If the compiler gets to the end of the three rules and there are still references for which it can’t figure out lifetimes, the compiler will stop with an error. These rules apply to fn definitions as well as impl blocks.



* The second rule is if there is exactly one input lifetime parameter, that lifetime is assigned to all output lifetime parameters

* The third rule is if there are multiple input lifetime parameters, but one of them is &self or &mut self because this is a method, the lifetime of self is assigned to all output lifetime parameters. 

    ```rs
    #![allow(unused_variables)]
    fn main() {
        struct ImportantExcerpt<'a> {
            part: &'a str,
        }

        impl<'a> ImportantExcerpt<'a> {
            fn announce_and_return_part(&self, announcement: &str) -> &str {
                println!("Attention please: {}", announcement);
                self.part
            }
        }
    }
    ```

#### Lifetime Annotations in Method Definitions

* Lifetime names for struct fields always need to be declared after the impl keyword and then used after the struct’s name, because those lifetimes are part of the struct’s type.

* In method signatures inside the impl block, references might be tied to the lifetime of references in the struct’s fields, or they might be independent. In addition, the lifetime elision rules often make it so that lifetime annotations aren’t necessary in method signatures.

    ```rs
    impl<'a> ImportantExcerpt<'a> {
        fn level(&self) -> i32 {
            3
        }
    }
    ```

#### The Static Lifetime

```rs
let s: &'static str = "I have a static lifetime.";
```


#### Generic Type, Trait Bounds & Lifetimes


```rs
use std::fmt::Display;

fn longest_with_an_announcement<'a, T>(x: &'a str, y: &'a str, ann: T) -> &'a str
    where T: Display
{
    println!("Announcement! {}", ann);
    if x.len() > y.len() {
        x
    } else {
        y
    }
}
```


#### Summary

Generic type parameters let you apply the code to different types. Traits and trait bounds ensure that even though the types are generic, they’ll have the behavior the code needs. You learned how to use lifetime annotations to ensure that this flexible code won’t have any dangling references. And all of this analysis happens at compile time, which doesn’t affect runtime performance!













<file_sep>+++

date = "2016-04-10T14:59:31+11:00"
title = "Azure Practices"
description = "Azure Practices"
+++


> *Here we are going to explore how to deploy web applications to Azure. From sep 2015, Microsoft launched new portal for Azure. To be honest, new portal is amazing, IMO, it is one of best changes from Microsoft.*

## Prerequisites

* You have experience with .net web applications.
* You have experience with website or web app deployment.

## Getting Started

* Register a Microsoft account. e.g. live.com, outlook.com, etc.
* Start free trial account on Azure cloud 
* Install Azure powershell, Azure CLI, Azure SDK for Visual Studio

## App Service

### Web app

*  Create a website from portal 

*  Create a website from visual studio 
  * Create Empty Asp.Net webstie with only index.html
  * Publish it to Azure via Web Deploy
    * Profile > Microsoft Azure App Service
    * Add Azure account > New app service ( name: webapp ) 
    * Publish method pick Web Deploy 
    * Validate Connection > Publish


*  Create a deploy slot ( webdeploy )
  *  Create a web app from portal. Default deployment slot is not available
  *  Change app service plan > Choose Standard tier 
  * Add slot > Type dev as name > leave the configuration plan as default
  * Change previous publish profile from Web Deploy to Web Deploy Package and create a package in local somewhere e.g. C:\webdeploy\webapp.zip

* Deploy to azure ( Start PowerShell as admin )

```
  Add-AzureAccount
  Get-AzureSubscription -Default `
  Publish-AzureWebsiteProject -Name 'webdeploy' -Package 'C:\webdeploy\webapp.zip'
```

* Deploy website to dev slot
**Publish... > Profile > Expand the webdeploy > Deployment slots > dev > Next > Publish



### Webjobs

* Open previous app service ( webapp )
* Set deployment credentials for Git / FTP
* Set Git Repo
  * Deploy options > Local Git Repository > Copy git URL https://xxxxxxx.git
  * Create demoWeb project with only one index page.
* demoWeb ( folder ) 
*#**  Index.html


* Deploy project to azure git repo ( Start powershell as admin)


```bash
  git init 
  git status 
  git config user.name "harryho" 
  git config user.email "<EMAIL>" 
  git commit -m "initial commit"
  git remote add azure https://xxxxxxx.git 
  git remote -v
  git push azure master 
```

* Use Kudu

 * DEVELOPMENT TOOLS > Advanced Tools

* Create a webjob and deploy to azure (Start powershell as admin) 

```bash
  mkdir webjob01
  cd webjob01
  echo "Get-Date | Out-File -FilePath 'd:\home\site\wwwroot\dateoutput.txt -Append' ">getdatejob.ps1
  .\getdatejob.ps1
  cat .\dateoutput.txt

  7z a -tzip getdatejob.zip *.ps1
  7z l getdatejob.zip

  help New-AzureWebisteJob
  New-AzureWebsiteJob -Name 'WebAppV1301' -JobName 'GetDateJob01'  -JobType Continuous -JobFile '.\getdatejob.zip'        

```
     
* Create webjob from visual studio
  * Create webjob project. Add one line program in the main method `Console.WriteLine("Hellow World"); `
  * Right click project node >   Publish Azure Webjob ...
  * Setup job schedule
  * Publish to app service ( webapp )


* Deploy Asp.net SPA with SQL database to Azure
  * Create blank SQL database from portal. Remember the admin Id and password
  * Install VS 2015, Azure SDK for VS 2015
  * Create web project ( c## ), choose ASP.NET Web Application ( .Net Framework )



### Custom Domain

* Free tier cannot custom domain name
* Bind the existing name  
* Navigate to Custom Domain

  * Copy the external IP ( e.g. 10.1.1.1 ) for later setup. 
  * Enter the domain name, validate domain name ( First time is invalid )
  * Open your domain register website, e.g. godaddy.com ( that is my domains register website)
  * Choose the domain you want to bind. Unlock the domain. 
  * Navigate to Zone tab 
  * Remove existing A type which points to hosting server
  * Add a new A Type pointing to azure 
  * Add an additional TXT Type pointing to azure for azure's verify
  * Save all changes and wait for DNS 
  * Use the site  https://digwebinterface.com to verify the new DNS been updated world wildly
  * Back to azure portal to validate the domain name. Once it is valid, save and update it.


### Self-signed SSL setup

* Create a text file named serverauth.cnf, then copy the following content into it, and then save it in a working directory. 

  ```ini
  [ req ]
  default_bits           = 2048
  default_keyfile        = privkey.pem
  distinguished_name     = req_distinguished_name
  attributes             = req_attributes
  x509_extensions        = v3_ca

  [ req_distinguished_name ]
  countryName         = Country Name (2 letter code)
  countryName_min         = 2
  countryName_max         = 2
  stateOrProvinceName     = State or Province Name (full name)
  localityName            = Locality Name (eg, city)
  0.organizationName      = Organization Name (eg, company)
  organizationalUnitName      = Organizational Unit Name (eg, section)
  commonName          = Common Name (eg, your app's domain name)
  commonName_max          = 64
  emailAddress            = Email Address
  emailAddress_max        = 40

  [ req_attributes ]
  challengePassword       = <PASSWORD>
  challengePassword_min       = 4
  challengePassword_max       = 20

  [ v3_ca ]
  subjectKeyIdentifier=hash
  authorityKeyIdentifier=keyid:always,issuer:always
  basicConstraints = CA:false
  keyUsage=nonRepudiation, digitalSignature, keyEncipherment
  extendedKeyUsage = serverAuth  
  ```

* In a command-line terminal, `CD` into your working directory and run the following command. Remember set your domain name as common name.

```
openssl req -sha256 -x509 -nodes -days 365 -newkey rsa:2048 -keyout myserver.key -out myserver.crt -config serverauth.cnf
```

* Export the certificate to a .pfx file by running the following command. When prompted, define a password to secure the .pfx file.

```
openssl pkcs12 -export -out myserver.pfx -inkey myserver.key -in myserver.crt
```

* Add SSL binding 
  * SNI SSL type for CNAME setup

* http to https redirect
  * DEVELOPMENT TOOLS > Advanced Tools > Debug Console> Powershell 
  * Navigate from site > wwwroot . 
  * Edit Web.config. Add url rewite into Web.config proply

```xml
<configuration>
  <system.webServer>
    <rewrite>
      <rules>
        <rule name="HTTP/S to HTTPS Redirect" enabled="true" stopProcessing="true">
        <match url="(.*)" />
        <conditions logicalGrouping="MatchAny">
          <add input="{SERVER_PORT_SECURE}" pattern="^1$" />
          <add input="{SERVER_PORT_SECURE}" pattern="^0$" />
        </conditions>
        <action type="Redirect" url="https://{HTTP_HOST}/OWA/" redirectType="Permanent" />
        </rule>
      </rules>
    </rewrite>
  </system.webServer>
</configuration>
```

* Restart the web.config (Optional ). Cleanup the browser cache and hard refresh





 

<file_sep>+++
title = "AWS: IAM"
weight = 1
description="IAM - Idenity and Acces Management: IAM Identity, Dos & Don'ts, Federation Integration"
+++


## IAM

AWS Identity and Access Management (IAM) enables you to manage access to AWS services and resources securely. Using IAM, you can create and manage AWS users and groups, and use permissions to allow and deny their access to AWS resources.


### Root User

* Every account has a root user in AWS. A root user is something that's created automatically for you whenever you create an AWS account.
* Every single AWS account has a root user. 
* The trouble is that root users have unrestricted access to every service and resource that is in AWS inside of your account. 
* The permissions of root user can't be restricted in any way. 


### Dos and Don'ts

* You should not be accessing the root account on a regular basis, whether that's daily, weekly, or whatever. 
* Make sure that you turn on multi-factor authentication on the root account. Multi-factor authentication used to be called two-factor authentication. It really just means that we know the password and we have some sort of a token that we will get a number generated. It's something that you might even use your smartphone for. But it now means that I have to know the username and password and I have to have this token that's going to generate a code. We'll see more about how you'll do that later. 
* Make sure that you've disabled your root access keys. This isn't the interactive login for root, it has to do with how we can access the account programmatically. 
* Make sure that you rotate the credentials. Just because we say don't log in doesn't mean set the password and then forget it. 
* Don't share the root user credentials. password. And all that the audit logs show is that root logged in and did the job. Kind of dangerous. 
* Make sure that you create a user that has administrative privileges that's assigned to you and that you know the password only.


### Features & Functions

* Allows user to have very secure access including through the use of multi-factor authentication and federation. 
* Grant user a lot of granular control over the specific resources.
* Grant temporary access to different people. 
* Simplify the number of logins by using federating identities
* Integrate the IAM solution is with all of the different products that AWS offers. 


### MFA

* MFA stands for multi-factor authentication
* Extra layer of security.
* Prevent against imposters, somebody who just happened to guess the right password or happened to actually shoulder surf and watch somebody key in their username and password.


### IAM User

___It is an entity that you create in AWS. The IAM user represents the person or service who uses the IAM user to interact with AWS. A primary use for IAM users is to give people the ability to sign in to the AWS Management Console for interactive tasks and to make programmatic requests to AWS services using the API or CLI. A user in AWS consists of a name, a password to sign into the AWS Management Console, and up to two access keys that can be used with the API or CLI.___

* Receive its own unique identifier, aka, ARN, which is an acronym, of course, which stands for Amazon Resource Name. Now, each one of these user accounts will have its own set of unique credentials, which again might be a common-sense factor, but still everybody needs their own username and password.
* Access the appropriate resources by using policies. 
* very easy to remove.
* set up specific permissions. It's a good practice.

### IAM Policy



* A policy is just a JSON-formatted document. It details what you can do, and it includes information from four different categories. 
* We can attach the policy to IAM identities. And those identities could be IAM user accounts, group accounts, or IAM roles.


### Federation

#### IAM Role

___It is very similar to a user, in that it is an identity with permission policies that determine what the identity can and cannot do in AWS. However, a role does not have any credentials (password or access keys) associated with it.___

* A role is used to assisgn temporary permissions
* Polices are associated to IAM Roles
* Roles are assumed by people or applications

#### IAM Group

___It is a collection of IAM users. You can use groups to specify permissions for a collection of users, which can make those permissions easier to manage for those users.___


#### Cross Accounts

* Flexible management allows me to take multiple accounts

* It's all token based, so it really optimizes the whole process of setting up security. 

* Manage a lot of different services between multiple accounts, as we as between business units

* Steps:
    * There are two different AWS accounts. There is one user in one account and we have some resources in another. 
    * Create a role that exists within the account with the resources.
    * create a role, that permits access to the resources, can be then applied to the person or group over in the account by the virtue of what we call the sts:AssumeRole, aka, temporary security credential, aka, .

* Flow chart

{{<mermaid>}}
graph BT
    EC2(Elastice Computer Cloud)
    S3(Simple Storage Service)
    RDS(Rational Database Service)
    temporay_security_credential-->IAM_User
    IAM_Role --> temporay_security_credential([sts:AssumeRole])

   temporay_security_credential;
   subgraph Account_1     
        IAM_User(IAM User or Group ABC)
        IAM_User --> EC2
        IAM_User --> S3
        IAM_User --> RDS
    end 
    subgraph Accuont_2 
        IAM_Role(IAM Role with IAM Policies attached)
        IAM_Role --> EC2
        IAM_Role --> S3
        IAM_Role --> RDS
    end
{{</mermaid >}}

* Samples:

The role `sts:AssumeRole` will be attached the IAM User or Group in Account 2

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::*:role/<Role_Of_Account_1>"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "sts:GetSessionToken",
            "Resource": "*"
        }
    ]
}
```

#### SAML

* SAML 2.0 is the one that you'll use for things like Active Directory. It stands for Security Assertion Markup Language. 

* It's an open standard. The idea is to create an open-standards-based approach that works with as many identity providers (idPs) as possible, things like Auth0, Microsoft Active Directory, and Shibboleth, etc. 

* Use existing corporate credentials for authentication and authorizaion in AWS console, CL, or API calls.

* Steps:

    * set up a SAML 2.0 -based federation,
    * Permit given identity to log in to the console, use CLI commands, or API calls. Now, this requires you to have an identity provider, such as Active Directory. 
    * when the user authenticates with Active Directory, Active Directory issues the SAML assertion document that says yep you are who you said you are and passes that over so that now when you go to access resources AWS says oh yeah
    * User is going to log in again to this identity provider. This checks against the local LDAP identity store. so again, Active Directory. 
    * Get confirmation, A token will be issue, that allows us to talk to Amazon's security token service, or STS, that allows us to assume a particular role within AWS. 
    * This passes back the temporary credentials. User is able to authenticate and gain access to acces AWS resources.
    * don't even have an AWS account.

#### AWS DS

* There are three different varieties. 
* Simple Active Directory for smaller, limited use cases.
* AWS Directory Service for Microsoft Active Directory, it's a mouthful. That's a fully-featured Active Directory. 
* Active Directory Connector that actually just facilitates or front ends the authentication request. And then the actual authentication is sent back across your VPN to your on-premises.




<file_sep>+++
title = "AWS : CLI - 3"
description = "AWS CLI & Security Group"
weight=2
+++


### AWS CLI & & Security Group

Sometimes it is so annoying to update the rules of security group one by one, because of the change of your public IP address. Following is a script to make such change easier.

The script will only update the SSH / RDP protocals of specified the security groups. The SSH and RDP are most popular ones which allow admin to access the remote EC2.

* Preparation 

  - Update **OLD_IPS** with your old IP address
  
  - Update **RDP_SG_LIST** and **SSH_SG_LIST** with your actual secuirty group IDs
  
  - Update **PROFILE** if your default profile is different


```bash
# Profile 
PROFILE=default

# Log file 
LOG=aws_sg.log

# Old IP list - Update your old IP address here.
OLD_IPS=(
  10.100.0.0
)

__show_sg_ids() {
    aws ec2 --profile $PROFILE describe-security-groups \
        --output json \
        --filters "Name=group-name,Values=*Bastion*" \
        --query 'SecurityGroups[*].{Name:GroupName,ID:GroupId,permissions:IpPermissions[*]}' | jq
}

__get_perm() {
    PROTOCOL=$1
    if [[ $PROTOCOL == "ssh" ]]; then
        PERM='[{"IpProtocol":"tcp","FromPort":22,"ToPort":22,"IpRanges":[{"CidrIp":"IP_ADDRESS/32"}]}]'
    elif [[ $PROTOCOL == "rdp" ]]; then
        PERM='[{"IpProtocol":"tcp","FromPort":3389,"ToPort":3389,"IpRanges":[{"CidrIp":"IP_ADDRESS/32"}]}]'
    fi
    echo $PERM
}

__get_desc() {
    PROTOCOL=$1
    if [[ $PROTOCOL == "ssh" ]]; then
        DESC='[{"IpProtocol":"tcp","FromPort":22,"ToPort":22,"IpRanges":[{"CidrIp":"IP_ADDRESS/32","Description":"Harry"}]}]'
    elif [[ $PROTOCOL == "rdp" ]]; then
        DESC='[{"IpProtocol":"tcp","FromPort":3389,"ToPort":3389,"IpRanges":[{"CidrIp":"IP_ADDRESS/32","Description":"Harry"}]}]'
    fi
    echo $DESC
}


__show_ips() {
    aws ec2 --profile $PROFILE \
        describe-security-groups \
        --output json \
        --query 'SecurityGroups[*].{Name:GroupName,ID:GroupId,permissions:IpPermissions[*]}' | grep -i "Harry" -C 2
}

__update_sg() {
    PROTOCOL=$1
    SGID=$2
    OIP=$3
    NIP=$4

    echo $PROFILE $SG $OIP $NIP | tee -a $LOG

    PERM=$(__get_perm $PROTOCOL)
    DESC=$(__get_desc $PROTOCOL)

    OLD_PERM=${PERM/"IP_ADDRESS"/$OIP}
    NEW_PERM=${PERM/"IP_ADDRESS"/$NIP}
    NEW_DESC=${DESC/"IP_ADDRESS"/$NIP}

    echo $OLD_PERM | tee -a $LOG

    echo $NEW_PERM | tee -a $LOG
    echo $NEW_DESC | tee -a $LOG

    aws ec2 --profile $PROFILE \
        revoke-security-group-ingress \
        --group-id $SGID --ip-permissions $OLD_PERM

    aws ec2 --profile $PROFILE \
        authorize-security-group-ingress \
        --group-id $SGID \
        --ip-permissions $NEW_PERM

    aws ec2 --profile $PROFILE \
        update-security-group-rule-descriptions-ingress \
        --group-id $SGID --ip-permissions $NEW_DESC

    aws ec2 --profile $PROFILE \
        describe-security-groups \
        --output json \
        --group-ids $SGID | jq

}

# Update the rule with RDP 
update_rdp_sg() {
    OIP=$1
    NIP=$2

    RDP_SG_LIST=(
        sg-0123456789 
        sg-9876543210
    )
    echo " :::::::::::: PROFILE - rdp :::::::::::: " | tee -a $LOG
    for SG in "${RDP_SG_LIST[@]}"; do
        __update_sg rdp $SG $OIP $NIP
    done
    __show_ips
}

# Update the rule with SSH 
update_ssh_sg() {
    OIP=$1
    NIP=$2

    SSH_SG_LIST=(
        sg-aaaaaaaaaaa 
        sg-bbbbbbbbbbb
    )
    echo " :::::::::::: PROFILE - ssh :::::::::::: " | tee -a $LOG
    for SG in "${SSH_SG_LIST[@]}"; do
        __update_sg ssh $SG $OIP $NIP
    done
    __show_ips
}


main() {
    echo 'Start...' $(date) | tee -a $LOG

    PROFILE=$1

    echo "profile $PROFILE " | tee -a $LOG

    echo 'You can pass profile name as 1st parameter to overwrite the default setting.'

    for OLD_IP in ${OLD_IPS[@]}; do
        NEW_IP=$(curl ifconfig.me)

        echo Old IP $OLD_IP | tee -a $LOG
        echo New IP $NEW_IP | tee -a $LOG

        # Update RDP bastion
        update_rdp_sg $OLD_IP $NEW_IP

        # Update SSH bastion
        update_ssh_sg $OLD_IP $NEW_IP
    done
    echo "DONE $(date) !!!!!!!!!! " | tee -a $LOG

}

main $@

```

* How to use 
 > ./update_sg.sh <profile_name>

```
./update_sg.sh profile_A 
./update_sg.sh profile_B 
```







<file_sep>+++


date = "2016-01-10T14:59:31+11:00"
title = "Ubuntu 14 -- desktop, extra tools"
description = "Post-installation for Ubuntu 14 desktop - Part 2"
draft = false
+++

## Prerequisites

* Install Ubunt 14 Desktop
* Internet is availble

## Install chrome

```bash
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo dpkg -i google-chrome-stable_current_amd64.deb
```


## Install Umake

```bash
sudo add-apt-repository ppa:ubuntu-desktop/ubuntu-make
sudo apt-get update
sudo apt-get install ubuntu-make
```

## Install IDEs via Umake

```bash
umake ide pycharm
umake web visual-studio-code
umake android android-studio
```



## General prerequest

```bash
sed -i "/^## deb .*partner/ s/^## //" /etc/apt/sources.list && apt-get update
sudo apt-get install geany byobu p7zip-full gimp pdfshuffler scribus \
filezilla lftp ubuntu-restricted-extras vlc pyrenamer \
imagemagick hugin darktable skype avidemux
```

## Remove Games

```bash

sudo apt-get remove aisleriot gnome-mahjongg gnomine gnome-sudoku

```

##  Geany themes
```bash
cd ~/Downloads
git clone https://github.com/codebrainz/geany-themes.git
mkdir ~/.config/geany/colorschemes
cp ~/Downloads/geany-themes/colorschemes/* ~/.config/geany/colorschemes/
rm -rf ~/Downloads/geany-themes
```

## Cloud
> from: http://www.webupd8.org/2014/06/install-copycom-client-in-ubuntu-or.html

```bash
sudo add-apt-repository ppa:paolorotolo/copy
sudo apt-get update
sudo apt-get install copy
sudo /opt/copy-client/CopyAgent -installOverlay
nautilus -q
copy
```
 
## Data processing

```bash
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E084DAB9
sudo add-apt-repository 'deb http://star-www.st-andrews.ac.uk/cran/bin/linux/ubuntu trusty/'
sudo apt-get update
sudo apt-get install spyder python-numpy python-numpy-doc sqlite3 \
python-scipy python-matplotlib python-matplotlib-doc r-base git-core
```

## Don't forget to use your own name and email!

```bash
git config --global user.name "<NAME>"
git config --global user.email "<EMAIL>"
```

## Maps and GIS software

```bash
sudo apt-get install python-software-properties
sudo add-apt-repository 'deb http://qgis.org/debian trusty main'
gpg --keyserver keyserver.ubuntu.com --recv DD45F6C3
gpg --export --armor DD45F6C3 | sudo apt-key add -
sudo apt-get update
sudo apt-get install qgis python-qgis qgis-plugin-grass grass-gui grass-doc \
libgdal1-dev libproj-dev gpsbabel
```

## Latex type stuff

```bash
sudo apt-get install jabref ibus-qt4 texlive texlive-latex-extra \
texlive-humanities texlive-fonts-extra latex-beamer
sudo apt-get -f install
```

## Package download and install (Texmaker and RStudio)

```bash
wget http://www.xm1math.net/texmaker/texmaker_ubuntu_14.04_4.4.1_amd64.deb
wget http://download1.rstudio.org/rstudio-0.98.1102-amd64.deb
sudo dpkg -i *.deb
sudo rm *.deb
 
sudo apt-get update && apt-get upgrade
sudo apt-get autoremove
```


```bash
sudo nano /etc/update-manager/release-upgrades
```
<file_sep>+++
date = "2018-12-04T14:59:31+11:00"
title = "SFTP & GPG "
description = "SFTP & GPG"
+++


### SFTP

SFTP (SSH File Transfer Protocol) is a secure file transfer protocol. It runs over the SSH protocol. It supports the full security and authentication functionality of SSH.

SFTP has pretty much replaced legacy FTP as a file transfer protocol, and is quickly replacing FTP/S. It provides all the functionality offered by these protocols, but more securely and more reliably, with easier configuration. There is basically no reason to use the legacy protocols any more.

SFTP also protects against password sniffing and man-in-the-middle attacks. It protects the integrity of the data using encryption and cryptographic hash functions, and autenticates both the server and the user.

#### Login SFTP with pass phrase

* Install expect 

```
# Ubuntu
sudo apt install expect 

# RH/CentOS
sudo yum install expect
```

* Set passphrase to global variable

```
export PASSPHRASE=<PASSWORD>
```

* Create a script - sftp.sh 

```bash

expect -c "
spawn sftp  -oPORT=9022 -oIdentityFile=~/.ssh/Your_SSH_Private_Key  -oPasswordAuthentication=no USER_ID@your.sftp.server.com
expect \"*\"
expect \"*\"
expect \"*\"
expect -nocase \"*passphrase*\" { send \"$PASSPHRASE\r\"; interact }
"
```

### GPG

GnuPG is a complete and free implementation of the OpenPGP standard as defined by RFC4880 (also known as PGP). GnuPG allows you to encrypt and sign your data and communications; it features a versatile key management system, along with access modules for all kinds of public key directories. GnuPG, also known as GPG, is a command line tool with features for easy integration with other applications. A wealth of frontend applications and libraries are available. GnuPG also provides support for S/MIME and Secure Shell (ssh).


#### Generate GPG key pair

* Generate key pair 
  
```
# Open terminal & run command below
gpg --gen-key

# Select RSA from options

Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)

# Use 4096
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096

# Pick option 0  
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years

# Enter user id, email and comment
GnuPG needs to construct a user ID to identify your key.

Real name: [Your_User_ID]
Email address: [<EMAIL>]
Comment: {Your_comment]

# Enter passphase & confirm it
You need a passphase to protect your secret key

Enter passphase: [your_passphase]
Repeat passphase: [your_passphase]


```


#### Export public key

* The third party requires your public key to decrypt your message, so you need to export public key.

```
gpg --armor --output [export_file_name]  --export [your_user_id]

## get finger print 
ggp --fingerprint

```

* Pass your public key and fingerprint to the third party

#### Import public key 


* Import public key file

```
gpg --import [public_key_file]
```

* Sign the public key with your priviate key


```
## Get user id of public key
gpg --list-keys

gpg --sign-key [public_key_user_id]

## Confirm to sign
Really sgin? (Y/N) Y

## Enter passphase 
Enter passphase: [your_pass_phase]

```

* Update the trust level 

```
gpg --edit-key [public_key_user_id]

## Enter trust after the command prompt
Command> trust

## Choose option 5 
Please decide how far you trust this user to correctly
verify other users' keys (by looking at passports,
checking fingerprints from different sources...)?

 1 = I don't know or won't say
 2 = I do NOT trust
 3 = I trust marginally
 4 = I trust fully
 s = I trust ultimately
 m = back to the main menu

Your decision? 5

# Enter q to quit
Command> q

```


#### PGP Encryption

* Encrypt a file

```
gpg --armor --encrypt \
 --recipient [public_key_user_id] --sign --local-user [your_user_id] \
 --output [encrypted_filename]

## Enter passphase
Enter passphase: [your_passphase]
```

#### PGP Decryption

* Decrypt a file


```
gpg --armor [decrpyted_filename] --decrypt [encrypted_filename]

## Enter passphase
Enter passphase: [your_passphase]
```<file_sep>+++
title = "AWS Note - 4"
description = "Reset EC2 password"
draft="true"
+++


## Reset EC2 password

Amazon Virtual Private Cloud (Amazon VPC) enables you to launch AWS resources into a virtual network that you've defined. This virtual network closely resembles a traditional network that you'd operate in your own data center, with the benefits of using the scalable infrastructure of AWS.

### EC2 with Linux



### EC2 with Windows 



<file_sep>+++
date = "2016-11-15T16:56:21+11:00"
title = "Angularjs Webpack ES6 Starter"
description = "This starter was inspired by another similar angular webpack starter repository"


+++

## Summary

It simply includes font-awesome, bootstrap for the people who don't want to use boostrap-webpack, font-awesome-webpack. I find it saves us so much effort to create prototype, since we don't need spectacular UI.


* This starter uses angular 1.5 for someone want to build component.
* This repo follows mvc patterns instead of component pattern. 
* ES6, and ES7 support with babel.
* Development server with live reload.
* Production builds with cache busting.
* Testing environment using karma to run tests and jasmine as the framework.
* Code coverage when tests are run.
* Include font-awesome without font-awesome-loader.
* Include Bootstrap 3 without bootstrap-loader.
* No gulp and no grunt, just npm scripts.

## Structure of starter

```bash

\path\to\angularjs-webpack-es6-starter
|   .babelrc                                    <-// default setting es2015.
|   karma.conf.js                               <-// tests and report setup 
|   webpack.config.js                           <-// webpack config
\---src
    |   tests.webpack.js
    |   
    +---app
    |   |   app.html                            <-// app view
    |   |   app.js                              <-// app module
    |   |   app.routes.js                       <-// app route to manage all routes 
    |   |   app.runner.js                       <-// app runner for state change enhancement  
    |   |   app.spec.js                         <-// app spec file for testing 
    |   |   
    |   +---common                              <-// common module for whole app
    |   |   |   common.js
    |   |   |   common.spec.js
    |   |   +---directives                      <-// common directives for whole app
    |   |   |       appUiDirectives.js
    |   |   |       appUiDirectives.spec.js
    |   |   |       commonDirectives.js
    |   |   |       commonDirectives.spec.js
    |   |   +---services                        <-// common views for whole app
    |   |   |       ApiService.js
    |   |   |       ApiService.spec.js
    |   |   |       UtilService.js
    |   |   |       UtilService.spec.js
    |   |   \---views                           <-// contains common views
    |   |           footer-view.html
    |   |           header-view.html
    |   |           sidebar-view.html
    |   |           topbar-view.html
    |   \---main                                <-// built-in fonts, css, images 
    |       \---dashboard
    |           +---controllers
    |           |       dashboardController.js
    |           |       dashboardController.spec.js
    |           \---views
    |                   dashboard-view.html
    |                   
    +---public                                  <-// built-in fonts, css, images 
    |   |   index.html
    |   +---fonts
    |   |   +--- ...
    |   \---img
    |       +--- favicon.ico
    \---style                                  <-// css files including customized css
           
    
```


## Browse [Repository](https://github.com/harryho/angularjs-webpack-es6-starter.git)



<file_sep>+++
date = "2017-02-03T10:59:31+11:00"
title = "CentOS 7 Server"
description = "CentOS 7 Server note"
draft = false
+++

Prelude

> This article is mainly to help experienced user install and setup CentOS 7 Server.


## Prerequisites

* You are familiar with CentOS, at least you have some experience working on Linux system. 
* You are familiar with basic bash/shell command


## Things to do after installing CentOS server

* How to setup your server 

## Firewall setup

```bash
sudo firewall-cmd --permanent --add-port=22/tcp
sudo firewall-cmd --permanent --add-port=21/tcp
sudo firewall-cmd --permanent --add-port=80/tcp
sudo firewall-cmd --permanent --add-port=443/tcp
sudo firewall-cmd --permanent --add-port=8080/tcp

sudo firewall-cmd --reload
```

## SSH server setup

`!!! For production environment, SSH should be secured by the CA`

* Install SSH if it is not done yet

```bash
## yum install openssh openssh-server openssh-clients openssl-libs
```

* Configure SSH

```bash
## backup default config 
sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.ori
sudo chmod a-w /etc/ssh/sshd_config.ori

## use any editor to update sshd_config 
sudo vi /etc/ssh/sshd_config 

## uncomment PasswordAuthentication yes to allow remote password login
## Password authentication is only for test environment

## setup ssh auto-start onboot
sudo systemctl restart sshd 
```

## Update Time Zone if it is incorrect

```bash
ls -l /etc/localtime ## check the time zone

sudo timedatectl list | grep New_York ## find the time zone by the city
sudo timedatectl set-timezone America/New_York
```


## Install Git

* Option 1: You can use `yum` to install git, but it is quite out-of-date. The version of git is 1.8.x

```bash
sudo yum install git
git --version
```

* Option 2: Download the latest stable release of Git and compile the software from source. (__Recommended__)

### Install build tools

```
sudo yum groupinstall "Development Tools"

sudo yum install gettext-devel openssl-devel perl-CPAN perl-devel \
    zlib-devel libcurl-devel expat-devel

sudo yum install yum-utils
```

### Download the latest release

```
wget https://github.com/git/git/archive/v2.x.x.tar.gz -O git.tar.gz
tar -zxf git.tar.gz
cd git-*
make configure
./configure --prefix=/usr/local
sudo make install
git --version
```



## Setup a better Vim

```bash
sudo yum isntall vim-enhanced

```


## Install Tmux

```bash
sudo yum install tmux
```


* Most useful tmux commands

> Ctrl+b " — split pane horizontally.
> 
> Ctrl+b % — split pane vertically.
> 
> Ctrl+b arrow key — switch pane.
> 
> Hold Ctrl+b, don’t release it and hold one of the arrow keys — resize pane.
> 
> Ctrl+b c — (c)reate a new window.
> 
> Ctrl+b , — rename reate a new window.
> 
> Ctrl+b n — move to the (n)ext window.
> 
> Ctrl+b p — move to the (p)revious window.


## Install python 3

You will only find Python 2 on CentOS by default. In order to install the latest python3, we need to install `IUS` to which stands for Inline with Upstream Stable. 

```bash
sudo yum -y install https://centos7.iuscommunity.org/ius-release.rpm
sudo yum -y install python36u

### Install development package
sudo yum -y insall python-devel python36u-devel
```


## Install nodejs

* Nodejs 6.x

```bash
sudo yum -y install nodejs
```

* Nodejs 8.x

```bash
curl --silent --location https://rpm.nodesource.com/setup_8.x | sudo bash -
```

* Upgrade NPM

```
sudo npm install -g npm
sudo npm install -g typescript
sudo mpm install -g yarn
```


## install docker CE (CentOS 7)

```bash
## add repo
sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

## check docker.list
yum list docker-ce --showduplicates | sort -r

## install docker engine
sudo yum install docker-ce

docker -v 
```


## Install JDK 8

* Downlaod the JDK from Oracle website. 

```bash

## At the confirmation prompt, enter y 
## then RETURN to continue with the installation.
sudo yum install java-1.8.0-openjdk-devel


java -version
```

* Test JDK with a simple HelloWorld program

```java
import java.util.Calendar;

class HelloWorld {
    public static void main(String[] args) {
        Calendar cal = Calendar.getInstance();
        int year = cal.get(Calendar.YEAR);
        int month = cal.get(Calendar.MONTH) + 1;
        int day = cal.get(Calendar.DATE);
        int hour = cal.get(Calendar.HOUR_OF_DAY);
        int minute = cal.get(Calendar.MINUTE);
        String username = System.getProperty("user.name");
        System.out.println(username+ ": Hello World! ");
        System.out.println(year + "/" + month + "/" + day + " " + hour + ":" + minute);
    }
}
```

* Compile and run the program

```bash
javac HelloWorld.java
java HelloWorld.java
```



## Install Go

* Install Go 

```bash
cd /tmp
curl -LO https://redirector.gvt1.com/edgedl/go/go1.9.2.linux-amd64.tar.gz

## check hash
shasum -a 256 go*linux-amd64.tar.gz

## install tar ball
sudo tar -C /usr/local -xvzf go1.9.2.linux-amd64.tar.gz
```

* Setup GOROOT 

```bash
cd /etc/profile.d
## Create a path.sh script
sudo vi path.sh
```

* Copy following code into `path.sh`

```
export PATH=$PATH:/usr/local/go/bin
```

* Setup local GOBIN, GOPATH

```bash
export GOBIN="$HOME/projects/go/bin"
export GOPATH="$HOME/projects/go/src"
export PATH
```


* Create a simple `hello.go` file to test 

```go
package main                                                        
                                                                    
import (                                                            
    "fmt"                                                           
    "log"                                                           
    "os/user"                                                       
)                                                                   
                                                                    
func main(){                                                        
    user, err := user.Current()                                 
    if err != nil {                                             
            log.Fatal(err)                                      
    }                                                            
    fmt.Printf(user.Name + " said : Hello World! \n" )            
}                                                                   
```

* Run the program

```bash
go run $GOPATH/hello.go
go install $GOPATH/hello.go
$GOBIN/hello
```



## Install Cmake

```bash
sudo yum install epel-release
sudo yum install cmake3
sudo ln -s /usr/bin/cmake3 /usr/bin/cmake
```


## Install Rust

```bash
curl -f -L https://static.rust-lang.org/rustup.sh -O
sh rustup.sh
rustc --version
```

## Install PHP 7

*  install and enable EPEL and Remi repository

```bash
sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo yum install http://rpms.remirepo.net/enterprise/remi-release-7.rpm
```

* install `yum-utils`

```
sudo yum install yum-utils
```

* Enable PHP 7 repo

```
sudo yum-config-manager --enable remi-php72  
```

* Install PHP

```
sudo yum install php php-mcrypt php-cli php-gd php-curl php-mysql \
    php-ldap php-zip php-fileinfo 
```


## Install clang

```bash
sudo yum install llvm
sudo yum install clang

```
<file_sep>package main

import (
	_ "bufio"
	_ "bytes"
	_ "compress/gzip"
	_ "encoding/gob"
	_ "flag"
	"fmt"
	_ "io/ioutil"
	"math"
	"os"
	_ "os"
	"reflect"
	_ "strings"
)

// func main() {
// 	fmt.Println("hello world")

// 	// var arr1 [5]int;

// 	// for  i := range arr1 {
// 	//     fmt.Printf(" index = %d , value = %d \n", i, arr1[i])
// 	// }

// 	// var arr2 = [...]int{0,0,0,0,0}
// 	// for  i := range arr2{
// 	//     fmt.Printf(" index = %d , value = %d \n", i, arr1[i])
// 	// }

// 	arr := [5]int{1, 2, 3, 4, 5}
// 	slice := arr[0:2]
// 	fmt.Printf(" slice = %v \n", slice)
// 	slice = arr[0:]
// 	fmt.Printf(" slice = %v \n", slice)
// 	slice = arr[:2]
// 	fmt.Printf(" slice = %v \n", slice)
// 	slice = arr[1:5]
// 	fmt.Printf(" slice = %v \n", slice)
// 	slice = arr[:5]
// 	fmt.Printf(" slice = %v \n", slice)

// 	sl_from := []int{1, 2, 3}
// 	sl_to := make([]int, 10)
// 	n := copy(sl_to, sl_from)
// 	fmt.Println(sl_to)
// 	// output: [1 2 3 0 0 0 0 0 0 0]
// 	fmt.Printf("Copied %d elements\n", n) // n == 3
// 	sl3 := []int{1, 2, 3}
// 	sl3 = append(sl3, 4, 5, 6)
// 	fmt.Println(sl3)

// 	str := "golang"
// 	chars := []byte(str)
// 	chars[1] = '0'
// 	newStr := string(chars)
// 	fmt.Println(newStr)
// }

// type Stack struct {
// }

// func (*Stack) Pop(st []int) int {
// 	v := 0
// 	for ix := len(st) - 1; ix >= 0; ix-- {
// 		if v = st[ix]; v != 0 {
// 			st[ix] = 0

// 		}
// 	}
// 	return v
// }

// func main() {
//     result := 0
//     for i:=0; i <= 10; i++ {
//         result = fibonacci(i)
//         fmt.Printf("fibonacci(%d) is: %d\n", i, result)
//     }
// }

// func fibonacci(n int) (res int) {
// 	if n <= 1 {
// 		res = 1
// 	} else {
// 		res = fibonacci(n-1) + fibonacci(n-2)
// 	}
// 	return
// }

// func main() {
// 	// callback(1, Add)
// 	plus := func(x, y int) int { return x + y }
// 	fmt.Println(plus(1, 2)) // 3

// 	// invoke func immediatley
// 	val := func(x, y int) int { return x + y }(1, 2) // 3
// 	fmt.Println(val)
// }

func Add(a, b int) {
	fmt.Printf("The sum of %d and %d is: %d\n", a, b, a+b)
}
func callback(y int, f func(int, int)) {
	f(y, 2)
	// this becomes Add(1, 2)
}

type TagType struct { //tags
	field1 bool   "An important answer"
	field2 string "The name of the thing"
	field3 int    "How much there are"
}

func refTag(tt TagType, ix int) {
	ttType := reflect.TypeOf(tt)
	ixField := ttType.Field(ix)
	fmt.Printf("%v\n", ixField.Tag)
}

type innerS struct {
	in1 int
	in2 int
}
type outerS struct {
	b int
	c float32
	int
	// anonymous field
	innerS // anonymous field
}

func Func() {
	outer2 := outerS{6, 7.5, 60, innerS{5, 10}}
	fmt.Println("outer2 is: ", outer2)
}

type Shape interface {
	Area() float32
}
type Square struct {
	side float32
}

type Circle struct {
	radius float32
}

func (sq *Square) Area() float32 {
	return sq.side * sq.side
}

func (c *Circle) Area() float32 {
	return c.radius * c.radius * math.Pi
}

type Rectangle struct {
	length, width float32
}

func (r Rectangle) Area() float32 {
	return r.length * r.width
}

func classifier(items ...interface{}) {
	for i, x := range items {
		switch x.(type) {
		case bool:
			fmt.Printf("param #%d is a bool\n", i)
		case float64:
			fmt.Printf("param #%d is a float64\n", i)
		case int, int64:
			fmt.Printf("param #%d is an int\n", i)
		case nil:
			fmt.Printf("param #%d is nil\n", i)
		case string:
			fmt.Printf("param #%d is a string\n", i)
		default:
			fmt.Printf("param #%d’s type is unknown\n", i)
		}
	}
}

func modifyValByReflect() {
	var x float64 = 3.4
	v := reflect.ValueOf(x)                      // Pass value
	fmt.Println("settability of v:", v.CanSet()) // false
	v = reflect.ValueOf(&x)                      // Note: take the address of x.
	fmt.Println("type of v:", v.Type())          // float64
	fmt.Println("settability of v:", v.CanSet()) // false
	v = v.Elem()
	fmt.Println("The Elem of v is: ", v)         // <float64 Value>
	fmt.Println("settability of v:", v.CanSet()) // true
	v.SetFloat(3.1415)                           // this works!
	fmt.Println(v.Interface())
	fmt.Println(v) // <float64 Value>
}

// func main() {
// 	shapes := []Shape{Rectangle{5, 3}, &Square{5}, &Circle{5}}
// 	fmt.Println("Looping through shapes for area ...")
// 	for n, _ := range shapes {
// 		fmt.Println("Shape details: ", shapes[n])
// 		fmt.Println("Area of this shape is: ", shapes[n].Area())
// 	}

// 	var shape Shape
// 	shape = &Square{4}

// 	switch t := shape.(type) {
// 	case *Square:
// 		fmt.Printf("Type Square %T with value %v\n", t, t)
// 	case *Circle:
// 		fmt.Printf("Type Circle %T with value %v\n", t, t)
// 	case *Rectangle:
// 		fmt.Printf("Type Rectangle %T with value %v\n", t, t)
// 	case nil:
// 		fmt.Println("nil value: nothing to check?")
// 	default:
// 		fmt.Printf("Unexpected type %T", t)
// 	}
// }

// func main() {
// 	inputFile, inputError := os.Open("input.dat")
// 	if inputError != nil {
// 		fmt.Printf("An error occurred on opening the inputfile\n" +
// 			"Does the file exist?\n" +
// 			"Have you got acces to it?\n")
// 		return // exit the function on error
// 	}
// 	defer inputFile.Close()
// 	inputReader := bufio.NewReader(inputFile)
// 	for {
// 		inputString, readerError := inputReader.ReadString('\n')
// 		if readerError == io.EOF {
// 			return
// 		}
// 		fmt.Printf("The input was: %s", inputString)
// 	}
// }

// var (
// 	firstName, lastName, s string
// 	i                      int
// 	f                      float32
// 	input                  = "56.12 / 5212 / Go"
// 	format                 = "%f / %d / %s"
// )

// func main() {
// 	fmt.Println("Please enter your full name: ")
// 	fmt.Scanln(&firstName, &lastName)
// 	// fmt.Scanf("%s %s", &firstName, &lastName)
// 	fmt.Printf("Hi %s %s!\n", firstName, lastName) // Hi <NAME>
// 	fmt.Sscanf(input, format, &f, &i, &s)
// 	fmt.Println("From the string we read: ", f, i, s)
// 	// ouwtput: From the string we read: 56.12 5212 Go
// }

// func main() {
// 	inputFile, inputError := os.Open("hello.go")
// 	if inputError != nil {
// 		fmt.Printf("An error occurred on opening the inputfile\n" +
// 			"Does the file exist?\n" +
// 			"Have you got acces to it?\n")
// 		return // exit the function on error
// 	}
// 	defer inputFile.Close()
// 	inputReader := bufio.NewReader(inputFile)
// 	for {
// 		inputString, readerError := inputReader.ReadString('\n')
// 		if readerError == io.EOF {
// 			return
// 		}
// 		fmt.Printf("The input was: %s", inputString)
// 	}
// }

// func main() {
// 	inputFile := "hello.go"
// 	outputFile := "hello.go.txt"
// 	buf, err := ioutil.ReadFile(inputFile)
// 	if err != nil {
// 		fmt.Fprintf(os.Stderr, "File Error: %s\n", err)
// 		// panic(err.Error())
// 	}
// 	fmt.Printf("%s\n", string(buf))
// 	err = ioutil.WriteFile(outputFile, buf, 0x644)
// 	if err != nil {
// 		panic(err.Error())
// 	}
//  // diff hello.go hello.go.txt ## return nothing
// }

// func main() {
// 	file, err := os.Open("csv_data.txt")
// 	if err != nil {
// 		panic(err)
// 	}
// 	defer file.Close()
// 	var col1, col2, col3 []string
// 	for {
// 		var v1, v2, v3 string
// 		_, err := fmt.Fscanln(file, &v1, &v2, &v3)
// 		// scans until newline
// 		if err != nil {
// 			break
// 		}
// 		col1 = append(col1, v1)
// 		col2 = append(col2, v2)
// 		col3 = append(col3, v3)
// 	}
// 	fmt.Println(" Col 1 ",col1)
// 	fmt.Println(" Col 3 ",col2)
// 	fmt.Println(" Col 3 ",col3)
// }

// func main() {
// 	fName := "public.zip"
// 	var r *bufio.Reader
// 	fi, err := os.Open(fName)
// 	if err != nil {
// 		fmt.Fprintf(os.Stderr, "%v, Can’t open %s: error: %s\n", os.Args[0],
// 			fName, err)
// 		os.Exit(1)
// 	}
// 	fz, err := gzip.NewReader(fi)
// 	if err != nil {
// 		r = bufio.NewReader(fi)
// 	} else {
// 		r = bufio.NewReader(fz)
// 	}
// 	for {
// 		line, err := r.ReadString('\n')
// 		if err != nil {

// 			fmt.Println("Done reading file")
// 			os.Exit(0)
// 		}
// 		fmt.Println(line)
// 	}
// }

// func main() {
// 	outputFile, outputError := os.OpenFile("output.txt",
// 		os.O_WRONLY|os.O_CREATE, 0666)
// 	if outputError != nil {
// 		fmt.Printf("An error occurred with file creation\n")
// 		return
// 	}
// 	defer outputFile.Close()
// 	outputWriter := bufio.NewWriter(outputFile)
// 	outputString := "hello world!\n"
// 	for i := 0; i < 10; i++ {
// 		outputWriter.WriteString(outputString)
// 	}
// 	outputWriter.Flush()
// }

// func main() {
// 	os.Stdout.WriteString("hello, world\n")
// 	f, _ := os.OpenFile("output.txt", os.O_CREATE|os.O_WRONLY, 0)
// 	defer f.Close()
// 	for i := 0; i < 10; i++ {
// 		f.WriteString("hello world!\n")
// 	}
// }

// func main() {
// 	CopyFile("target_hello.txt", "hello.go")
// 	fmt.Println("Copy done!")

// }
// func CopyFile(dstName, srcName string) (written int64, err error) {
// 	src, err := os.Open(srcName)
// 	if err != nil {
// 		return
// 	}
// 	defer src.Close()
// 	dst, err := os.OpenFile(dstName, os.O_WRONLY|os.O_CREATE, 0644)
// 	if err != nil {
// 		return
// 	}
// 	defer dst.Close()
// 	return io.Copy(dst, src)
// }

// func main() {
// 	who := "Harry "
// 	if len(os.Args) > 1 {
// 		who += strings.Join(os.Args[1:], " ")
// 	}
// 	fmt.Println("Good Morning", who)
// }

// var NewLine = flag.Bool("n", false, "print on newline")

// // echo -n flag, of type *bool
// const (
// 	Space   = " "
// 	Newline = "\n"
// )

// func main() {
// 	flag.PrintDefaults()
// 	flag.Parse()
// 	var s string = ""
// 	for i := 0; i < flag.NArg(); i++ {
// 		if i > 0 {
// 			s += Space
// 		}
// 		s += flag.Arg(i)
// 	}
// 	if *NewLine { // -n is parsed, flag becomes true
// 		s += Newline
// 	}
// 	os.Stdout.WriteString(s)
// }

// func cat(r *bufio.Reader) {
// 	for {
// 		buf, err := r.ReadBytes('\n')
// 		if err == io.EOF {
// 			break
// 		}
// 		fmt.Fprintf(os.Stdout, "%s", buf)
// 	}
// 	return
// }

// func cat(f *os.File) {
// 	const NBUF = 512
// 	var buf [NBUF]byte
// 	for {
// 		switch nr, err := f.Read(buf[:]); true {
// 		case nr < 0:
// 			fmt.Fprintf(os.Stderr, "cat: error reading: %s\n", err.Error())
// 			os.Exit(1)
// 		case nr == 0: // EOF
// 			return
// 		case nr > 0:
// 			if nw, ew := os.Stdout.Write(buf[0:nr]); nw != nr {
// 				fmt.Fprintf(os.Stderr, "cat: error writing: %s\n",
// 					ew)
// 			}
// 		}
// 	}
// }

// func main() {
// 	flag.Parse()
// 	if flag.NArg() == 0 {
// 		cat(os.Stdin)
// 	}

// 	for i := 0; i < flag.NArg(); i++ {
// 		f, err := os.Open(flag.Arg(i))
// 		if err != nil {
// 			fmt.Fprintf(os.Stderr, "%s:error reading from %s: %s\n",
// 				os.Args[0], flag.Arg(i), err.Error())
// 			os.Exit(1)
// 		}
// 		cat(f)
// 		f.Close()
// 	}
// }

// func main() {
// 	// unbuffered: os.Stdout implements io.Writer
// 	fmt.Fprintf(os.Stdout, "%s\n", "hello world! - unbuffered")
// 	// buffered:
// 	buf := bufio.NewWriter(os.Stdout)
// 	// and now so does buf:
// 	fmt.Fprintf(buf, "%s\n", "hello world! - buffered")
// 	buf.Flush()
// }

// type Address struct {
// 	Type    string
// 	City    string
// 	Country string
// }
// type VCard struct {
// 	FirstName string
// 	LastName  string
// 	Addresses []*Address
// 	Remark    string
// }

// func main() {
// 	pa := &Address{"private", "Aartselaar", "Belgium"}
// 	wa := &Address{"work", "Boom", "Belgium"}
// 	vc := VCard{"Jan", "Kersschot", []*Address{pa, wa}, "none"}
// 	// fmt.Printf("%v: \n", vc)
// 	// {<NAME> [0x126d2b80 0x126d2be0] none}:
// 	// JSON format:
// 	js, _ := json.Marshal(vc)
// 	fmt.Printf("JSON format: %s", js)
// 	// using an encoder:
// 	file, _ := os.OpenFile("vcard.json", os.O_CREATE|os.O_WRONLY, 0)
// 	defer file.Close()
// 	enc := json.NewEncoder(file)
// 	err := enc.Encode(vc)
// 	if err != nil {
// 		fmt.Println("Error in encoding json")
// 	}
// }

// var t, token xml.Token
// var err error

// func main() {
// 	input :=
// 		"<Person><FirstName>Laura</FirstName><LastName>Lynn</LastName></Person>"
// 	inputReader := strings.NewReader(input)
// 	p := xml.NewDecoder(inputReader)
// 	for t, err = p.Token(); err == nil; t, err = p.Token() {
// 		switch token := t.(type) {
// 		case xml.StartElement:
// 			name := token.Name.Local
// 			fmt.Printf("Token name: %s\n", name)
// 			for _, attr := range token.Attr {
// 				attrName := attr.Name.Local
// 				attrValue := attr.Value
// 				fmt.Printf("An attribute is: %s %s\n", attrName,
// 					attrValue)
// 				// ...
// 			}
// 		case xml.EndElement:
// 			fmt.Println("End of token")
// 		case xml.CharData:
// 			content := string([]byte(token))
// 			fmt.Printf("This is the content: %v\n", content)
// 		// ...
// 		default:
// 			// ...
// 		}
// 	}
// }

// type P struct {
// 	X, Y, Z int
// 	Name    string
// }
// type Q struct {
// 	X, Y *int32
// 	Name string
// }

// func main() {
// 	// Initialize the encoder and decoder. Normally enc and dec would
// 	// be bound to network connections and the encoder and decoder
// 	// would run in different processes.
// 	var network bytes.Buffer
// 	// Stand-in for a network connection
// 	enc := gob.NewEncoder(&network) // Will write to network.
// 	dec := gob.NewDecoder(&network)
// 	// Will read from network.
// 	// Encode (send) the value.
// 	err := enc.Encode(P{3, 4, 5, "Pythagoras"})
// 	if err != nil {
// 		log.Fatal("encode error:", err)
// 	}
// 	// Decode (receive) the value.
// 	var q Q
// 	err = dec.Decode(&q)
// 	if err != nil {
// 		log.Fatal("decode error:", err)
// 	}
// 	fmt.Printf("%q: {%d,%d}\n", q.Name, *q.X, *q.Y)
// }

// type Address struct {
// 	Type    string
// 	City    string
// 	Country string
// }
// type VCard struct {
// 	FirstName string
// 	LastName  string
// 	Addresses []*Address
// 	Remark    string
// }

// func main() {
// 	pa := &Address{"private", "Aartselaar", "Belgium"}
// 	wa := &Address{"work", "Boom", "Belgium"}
// 	vc := VCard{"Jan", "Kersschot", []*Address{pa, wa}, "none"}
// 	// fmt.Printf("%v: \n", vc)
// 	// {<NAME> [0x126d2b80 0x126d2be0] none}:
// 	// using an encoder:
// 	file, _ := os.OpenFile("vcard.gob", os.O_CREATE|os.O_WRONLY, 0)
// 	defer file.Close()
// 	enc := gob.NewEncoder(file)
// 	err := enc.Encode(vc)
// 	if err != nil {
// 		log.Println("Error in encoding gob")
// 	}
// }

// func main() {
// 	hasher := sha1.New()
// 	io.WriteString(hasher, "test")
// 	b := []byte{}
// 	fmt.Printf("Result: %x\n", hasher.Sum(b))
// 	fmt.Printf("Result: %d\n", hasher.Sum(b))
// 	hasher.Reset()
// 	data := []byte("We shall overcome!")
// 	n, err := hasher.Write(data)
// 	if n != len(data) || err != nil {
// 		log.Printf("Hash write error: %v / %v", n, err)
// 	}
// 	checksum := hasher.Sum(b)
// 	fmt.Printf("Checksum: %x\n", checksum)
// }

//---deadlock -----
// func f1(in chan int) {
// 	fmt.Println(<-in)
// }
// func main() {
// 	out := make(chan int)
// 	out <- 2
// 	go f1(out)
// }

// func compute(ch chan int) {
// 	ch <- someComputation()
// 	// when it completes, signal on the channel.
// }

// func main() {
// 	ch := make(chan int) // allocate a channel.
// 	go compute(ch)       // start something in a goroutine
// 	doSomethingElseForAWhile()
// 	result := <-ch
// }

// func main() {
// 	runtime.GOMAXPROCS(4)
// 	go func() {
// 		for i := 0; i < 10; i++ {
// 			println("Hello")
// 			time.Sleep(500)
// 		}

// 	}()
// 	go func() {
// 		for i := 0; i < 10; i++ {
// 			println("World")
// 			time.Sleep(500)
// 		}
// 	}()
// 	time.Sleep(999999)
// }

// func main() {
// 	runtime.GOMAXPROCS(4)

// 	start := time.Now()

// 	stockSymbols := []string{
// 		"goog",
// 		"msft",
// 		"aapl",
// 		"bbry",
// 		"hpq",
// 		"vz",
// 		"t",
// 		"tmus",
// 		"s",
// 	}

// 	numComplete := 0

// 	for _, symbol := range stockSymbols {
// 		go func(symbol string) {
// 			resp, _ := http.Get("http://dev.markitondemand.com/Api/v2/Quote?symbol=" + symbol)
// 			defer resp.Body.Close()
// 			body, _ := ioutil.ReadAll(resp.Body)

// 			quote := new(QuoteResponse)
// 			xml.Unmarshal(body, &quote)

// 			fmt.Printf("%s: $%.2f\n", quote.Name, quote.LastPrice)
// 			numComplete++
// 		}(symbol)
// 	}

// 	for numComplete < len(stockSymbols) {
// 		time.Sleep(10 * time.Millisecond)
// 	}
// 	elapsed := time.Since(start)

// 	fmt.Printf("Execution Time: %s", elapsed)
// }

// type QuoteResponse struct {
// 	Status           string
// 	Name             string
// 	LastPrice        float32
// 	Change           float32
// 	ChangePercent    float32
// 	TimeStamp        string
// 	MSDate           float32
// 	MarketCap        int
// 	Volume           int
// 	ChangeYTD        float32
// 	ChangePercentYTD float32
// 	High             float32
// 	Low              float32
// 	Open             float32
// }

// func main() {
// 	ch := make(chan string, 2)
// 	ch <- "Hello"
// 	ch <- "Hello"
// 	close(ch)
// 	fmt.Println(<-ch)
// 	fmt.Println(<-ch)
// 	ch <- "Hello"
// }

// func main() {
// 	stream := pump()
// 	go suck(stream)
// 	// the above 2 lines can be shortened to: go suck( pump() )
// 	time.Sleep(1e9)
// }
// func pump() chan int {
// 	ch := make(chan int)
// 	go func() {
// 		for i := 0; ; i++ {
// 			ch <- i
// 		}
// 	}()
// 	return ch
// }
// func suck(ch chan int) {
// 	for {
// 		fmt.Println(<-ch)
// 	}
// }

// func main() {
// 	suck(pump())
// 	time.Sleep(1e9)
// }
// func pump() chan int {
// 	ch := make(chan int)
// 	go func() {
// 		for i := 0; ; i++ {
// 			ch <- i
// 		}
// 	}()
// 	return ch
// }
// func suck(ch chan int) {
// 	go func() {
// 		for v := range ch {
// 			fmt.Println(v)
// 		}
// 	}()
// }

// func (c *container) Iter() <-chan items {
// 	ch := make(chan item)
// 	go func() {
// 		for i := 0; i < c.Len(); i++ {
// 			// or use a for-range loop
// 			ch <- c.items[i]
// 		}
// 	}()
// 	return ch
// }

// func main() {
// 	sendChan := make(chan int)
// 	receiveChan := make(chan string)
// 	go processChannel(sendChan, receiveChan)

// }

// func processChannel(in <-chan int, out chan<- string) {
// 	for inValue := range in {
// 		result := strconv.Itoa(inValue) // processing inValue
// 		// ...
// 		out <- result
// 	}
// }

// // Send the sequence 2, 3, 4, ... to returned channel
// func generate() chan int {
// 	ch := make(chan int)
// 	go func() {
// 		for i := 2; ; i++ {
// 			ch <- i
// 		}
// 	}()
// 	return ch
// }

// // Filter out input values divisible by prime, send rest to returned channel
// func filter(in chan int, prime int) chan int {
// 	out := make(chan int)
// 	go func() {
// 		for {
// 			if i := <-in; i%prime != 0 {
// 				out <- i
// 			}
// 		}
// 	}()
// 	return out
// }

// func sieve() chan int {
// 	out := make(chan int)
// 	go func() {
// 		ch := generate()
// 		for {
// 			prime := <-ch
// 			ch = filter(ch, prime)
// 			out <- prime
// 		}
// 	}()
// 	return out
// }

// func main() {
// 	primes := sieve()
// 	for {
// 		fmt.Println(<-primes)
// 	}
// }

// func main() {
// 	runtime.GOMAXPROCS(2) // in goroutine_select2.go
// 	ch1 := make(chan int)
// 	ch2 := make(chan int)
// 	go pump1(ch1)
// 	go pump2(ch2)
// 	go suck(ch1, ch2)
// 	time.Sleep(1e9)
// }
// func pump1(ch chan int) {
// 	for i := 0; ; i++ {
// 		ch <- i * 2
// 	}
// }
// func pump2(ch chan int) {
// 	for i := 0; ; i++ {
// 		ch <- i + 5
// 	}
// }
// func suck(ch1 chan int, ch2 chan int) {
// 	for {
// 		select {
// 		case v := <-ch1:

// 			fmt.Printf("Received on channel 1: %d\n", v)
// 		case v := <-ch2:
// 			fmt.Printf("Received on channel 2: %d\n", v)
// 		}
// 	}
// }

// var resume chan int

// func integers() chan int {
// 	yield := make(chan int)
// 	count := 0
// 	go func() {
// 		for {
// 			yield <- count
// 			count++
// 		}
// 	}()
// 	return yield
// }
// func generateInteger() int {
// 	return <-resume
// }
// func main() {
// 	resume = integers()
// 	fmt.Println(generateInteger()) //=> 0
// 	fmt.Println(generateInteger()) //=> 1
// 	fmt.Println(generateInteger()) //=> 2
// }

// type Any interface{}
// type EvalFunc func(Any) (Any, Any)

// func main() {
// 	evenFunc := func(state Any) (Any, Any) {
// 		oldSate := state.(int)
// 		newState := oldSate + 2
// 		return oldSate, newState
// 	}
// 	even := BuildLazyIntEvaluator(evenFunc, 0)
// 	for i := 0; i < 10; i++ {
// 		fmt.Printf("%vth even: %v\n", i, even())
// 	}
// }

// func BuildLazyEvaluator(evalFunc EvalFunc, initState Any) func() Any {

// 	retValChan := make(chan Any)
// 	loopFunc := func() {
// 		var actState Any = initState
// 		var retVal Any
// 		for {
// 			retVal, actState = evalFunc(actState)
// 			retValChan <- retVal
// 		}
// 	}
// 	retFunc := func() Any {
// 		return <-retValChan
// 	}
// 	go loopFunc()
// 	return retFunc
// }

// func BuildLazyIntEvaluator(evalFunc EvalFunc, initState Any) func() int {
// 	evalFn := BuildLazyEvaluator(evalFunc, initState)
// 	return func() int {
// 		return evalFn().(int)
// 	}
// }

// type Request struct {
// 	a, b      int
// 	replyChan chan int // reply channel inside the Request
// }

// type binOp func(a, b int) int

// func run(op binOp, req *Request) {
// 	req.replyChan <- op(req.a, req.b)
// }

// func server(op binOp, service chan *Request, quitChan chan bool) {
// 	for {
// 		select {
// 		case req := <-service:
// 			go run(op, req)
// 		case <-quitChan:
// 			return
// 		}
// 	}
// }
// func startServer(op binOp) (service chan *Request, quitChan chan bool) {
// 	service = make(chan *Request)
// 	quitChan = make(chan bool)
// 	go server(op, service, quitChan)
// 	return service, quitChan
// }

// func main() {
// 	adder, quitChan := startServer(func(a, b int) int { return a + b })
// 	const N = 100
// 	var reqs [N]Request
// 	for i := 0; i < N; i++ {
// 		req := &reqs[i]
// 		req.a = i
// 		req.b = i + N
// 		req.replyChan = make(chan int)
// 		adder <- req
// 	}
// 	// checks:
// 	for i := N - 1; i >= 0; i-- { // doesn’t matter what order
// 		if <-reqs[i].replyChan != N+2*i {
// 			fmt.Println("fail at", i)
// 		} else {

// 			fmt.Println("Request ", i, "is ok!")
// 		}
// 	}
// 	quitChan <- true
// 	fmt.Println("done")
// }

// type Person struct {
// 	Name   string
// 	salary float64
// 	chF    chan func()
// }

// func NewPerson(name string, salary float64) *Person {
// 	p := &Person{name, salary, make(chan func())}
// 	go p.backend()
// 	return p
// }
// func (p *Person) backend() {
// 	for f := range p.chF {
// 		f()
// 	}
// }

// // Set salary.
// func (p *Person) SetSalary(sal float64) {
// 	p.chF <- func() { p.salary = sal }
// }

// // Retrieve salary.
// func (p *Person) Salary() float64 {
// 	fChan := make(chan float64)
// 	p.chF <- func() { fChan <- p.salary }
// 	return <-fChan
// }
// func (p *Person) String() string {
// 	return "Person - name is: " + p.Name + " - salary is: " + strconv.FormatFloat(p.Salary(), 'f', 2, 64)
// }
// func main() {
// 	bs := NewPerson("<NAME>", 2500.5)
// 	fmt.Println(bs)
// 	bs.SetSalary(4000.25)
// 	fmt.Println("Salary changed:")
// 	fmt.Println(bs)
// }

// var values = [5]int{10, 11, 12, 13, 14}

// func main() {
// 	// version A:
// 	fmt.Println("\nVersion A:")
// 	for ix := range values { // ix is the index
// 		func() {
// 			fmt.Print(ix, " ")
// 		}() // call closure, prints each index
// 	}
// 	fmt.Println()
// 	// version B: same as A, but call closure as a goroutine
// 	fmt.Println("\nVersion B:")
// 	for ix := range values {
// 		go func() {
// 			fmt.Print(ix, " ")
// 		}()
// 	}
// 	fmt.Println()
// 	time.Sleep(5e9)
// 	// version C: the right way
// 	fmt.Println("\n\nVersion C:")
// 	for ix := range values {
// 		go func(ix interface{}) {
// 			fmt.Print(ix, " ")
// 		}(ix)
// 	}
// 	fmt.Println()
// 	time.Sleep(5e9)
// 	// version D: print out the values:
// 	fmt.Println("\n\nVersion D:")
// 	for ix := range values {

// 		val := values[ix]
// 		go func() {
// 			fmt.Print(val, " ")
// 		}()
// 	}
// 	time.Sleep(1e9)
// }

// func badCall() {
// 	panic("bad end")
// }

// func test() {
// 	defer func() {
// 		if e := recover(); e != nil {
// 			fmt.Printf("Panicking %s\r\n", e)
// 		}
// 	}()
// 	badCall()
// 	fmt.Printf("After bad call\r\n")
// }

// func main() {
// 	fmt.Printf("Calling test\r\n")
// 	test()
// 	fmt.Printf("Test completed\r\n")
// }

// func main() {
// 	// 1) os.StartProcess //
// 	/*********************/
// 	/* Linux: */
// 	env := os.Environ()
// 	procAttr := &os.ProcAttr{
// 		Env: env,
// 		Files: []*os.File{
// 			os.Stdin,
// 			os.Stdout,
// 			os.Stderr,
// 		},
// 	}
// 	pid, err := os.StartProcess("/bin/ls", []string{"ls", "-l"}, procAttr)
// 	if err != nil {
// 		fmt.Printf("Error %v starting process!", err) //
// 		os.Exit(1)
// 	}
// 	fmt.Printf("The process id is %v", pid)
// 	/* Output:
// 	The process id is &{21275 0 0 {{0 0} 0 0 0 0}}The process id is &{21276 0 0 {{0 0} 0 0 0 0}}total 54
// 	-rwxrwxrwx 1 root root   250 Sep 21 19:33 csv_data.txt
// 	-rwxrwxrwx 1 root root 25227 Oct  4 23:34 hello.go
// 	-rwxrwxrwx 1 root root  6708 Sep 21 10:25 hello.go.txt
// 	-rwxrwxrwx 1 root root   130 Sep 21 11:08 output.txt
// 	-rwxrwxrwx 1 root root  8898 Sep 21 12:10 target_hello.txt
// 	-rwxrwxrwx 1 root root  1619 Sep 22 14:40 urlshorten.go.txt
// 	-rwxrwxrwx 1 root root   182 Sep 21 13:50 vcard.json
// 	*/
// 	// 2nd example: show all processes
// 	pid, err = os.StartProcess("/bin/ps", []string{"-e", "opid,ppid,comm"}, procAttr)
// 	if err != nil {
// 		fmt.Printf("Error %v starting process!", err) //
// 		os.Exit(1)
// 	}
// 	fmt.Printf("The process id is %v", pid)
// 	// 2) cmd.Run //
// 	/***************/
// 	cmd := exec.Command("gedit") // this opens a gedit-window
// 	err = cmd.Run()
// 	if err != nil {
// 		fmt.Printf("Error %v executing command!", err)
// 		os.Exit(1)
// 	}
// 	fmt.Printf("The command is %v", cmd)
// }

// const (
// 	AvailableMemory         = 10 << 20                                  // 10 MB, for example
// 	AverageAverageMemoryPerRequest = 10 << 10                                  // 10 KB
// 	MAXREQS                 = AvailableMemory / AverageAverageMemoryPerRequest // here amounts to 1000
// )

// var sem = make(chan int, MAXREQS)

// type Request struct {
// 	a, b   int
// 	replyc chan int
// }

// func process(r *Request) {
// 	// Do something
// 	// May take a long time and use a lot of memory or CPU
// }
// func handle(r *Request) {
// 	process(r)
// 	// signal done: enable next request to start
// 	// by making 1 empty place in the buffer
// 	<-sem
// }
// func Server(queue chan *Request) {
// 	for {
// 		sem <- 1
// 		// blocks when channel is full (1000 requests are active)
// 		// so wait here until there is capacity to process a request
// 		// (doesn’t matter what we put in it)
// 		request := <-queue
// 		go handle(request)
// 	}
// }
// func main() {
// 	fmt.Println(" AvailableMemory ", AvailableMemory)
// 	fmt.Println(" AverageAverageMemoryPerRequest ", AverageAverageMemoryPerRequest)
// 	queue := make(chan *Request)
// 	go Server(queue)
// }

// const NCPU = 4

// func DoAll() {
// 	sem := make(chan int, NCPU) // Buffering optional but sensible.
// 	for i := 0; i < NCPU; i++ {
// 		go DoPart(sem)

// 	}
// 	// Drain the channel sem, waiting for NCPU tasks to complete
// 	for i := 0; i < NCPU; i++ {
// 		<-sem // wait for one task to complete
// 	}
// 	// All done.
// }
// func DoPart(sem chan int) {
// 	// do the part of the computation

// 	sem <- 1 // signal that this piece is done
// }

// func main() {
// 	runtime.GOMAXPROCS(NCPU)
// 	DoAll()
// }

// type Person struct {
// 	Name   string
// 	salary float64
// 	chF    chan func()
// }

// func NewPerson(name string, salary float64) *Person {
// 	p := &Person{name, salary, make(chan func())}
// 	go p.backend()
// 	return p
// }
// func (p *Person) backend() {
// 	for f := range p.chF {
// 		f()
// 	}
// }

// // Set salary.
// func (p *Person) SetSalary(sal float64) {
// 	p.chF <- func() { p.salary = sal }
// }

// // Retrieve salary.
// func (p *Person) Salary() float64 {
// 	fChan := make(chan float64)
// 	p.chF <- func() { fChan <- p.salary }
// 	return <-fChan
// }
// func (p *Person) String() string {
// 	return "Person - name is: " + p.Name + " - salary is: " + strconv.
// 		FormatFloat(p.Salary(), 'f', 2, 64)
// }
// func main() {
// 	bs := NewPerson("<NAME>", 2500.5)
// 	fmt.Println(bs)
// 	bs.SetSalary(4000.25)
// 	fmt.Println("Salary changed:")
// 	fmt.Println(bs)
// }

// /* Output Person - name is: <NAME> - salary is: 2500.50
// Salary changed:
// Person - name is: <NAME> - salary is: 4000.25 *
// */

// func main() {
// 	fmt.Println("sync", testing.Benchmark(BenchmarkChannelSync).String())
// 	fmt.Println("buffered", testing.Benchmark(BenchmarkChannelBuffered).String())
// }

// func BenchmarkChannelSync(b *testing.B) {
// 	ch := make(chan int)
// 	go func() {
// 		for i := 0; i < b.N; i++ {
// 			ch <- i
// 		}
// 		close(ch)
// 	}()
// 	for _ = range ch {
// 	}
// }

// func BenchmarkChannelBuffered(b *testing.B) {
// 	ch := make(chan int, 128)
// 	go func() {
// 		for i := 0; i < b.N; i++ {
// 			ch <- i
// 		}
// 		close(ch)
// 	}()
// 	for _ = range ch {
// 	}
// }

// /* Output:
// Windows: N Time 1 op Operations per sec
// sync 1000000 2443 ns/op --> 409 332 / s
// buffered 1000000 4850 ns/op --> 810 477 / s
// Linux:
// */

func main() {
	file, err := os.Open("csv_data.txt")

	if err != nil {
		fmt.Printf("An error occurred on opening the inputfile\n" +
			"Does the file exist?\n" +
			"Have you got acces to it?\n")
		return
	}
	defer file.Close()
	cat(file)
	// iReader := bufio.NewReader(file)
	// for {
	// 	str, err := iReader.ReadString('\n')
	// 	if err != nil {
	// 		return // error or EOF
	// 	}
	// 	fmt.Printf("The input was: %s", str)
	// }
}

func cat(f *os.File) {
	const NBUF = 512
	var buf [NBUF]byte
	for {
		switch nr, er := f.Read(buf[:]); true {
		case nr < 0:
			fmt.Fprintf(os.Stderr, "cat: error reading from %s: %s\n", f, er)
			os.Exit(1)
		case nr == 0: // EOF
			return
		case nr > 0:
			if nw, ew := os.Stdout.Write(buf[0:nr]); nw != nr {
				fmt.Fprintf(os.Stderr, "cat: error writing from %s: %s\n", f, ew)
			}
		}
	}
}
<file_sep>+++
title = "AWS: VPC - 3"
description = "VPC Peering, Direct Connect, Transit Gateway"
weight=5

+++


## VPC Part 3


### Endpoint

A VPC endpoint enables you to privately connect your VPC to supported AWS services and VPC endpoint services powered by AWS PrivateLink without requiring an internet gateway, NAT device, VPN connection, or AWS Direct Connect connection.

Endpoints are virtual devices. They are horizontally scaled, redundant, and highly available VPC components. They allow communication between instances in your VPC and services without imposing availability risks or bandwidth constraints on your network traffic.



* Endpoint service — Your own application in your VPC. Other AWS principals can create a connection from their VPC to your endpoint service

* Gateway endpoint — A gateway endpoint is a gateway that you specify as a target for a route in your route table for traffic destined to a supported AWS service.

> Amazon S3

> DynamoDB


* Interface endpoint — An interface endpoint is an elastic network interface with a private IP address from the IP address range of your subnet that serves as an entry point for traffic destined to a supported service.


#### Limits of Interface Endpoint

* For each interface endpoint, you can choose only one subnet per Availability Zone.
* Interface endpoints support the use of policies for services that support endpoint policies.
* An interface endpoint supports TCP traffic only.
* Endpoints are supported within the same Region only. 
* Endpoints support IPv4 traffic only.



#### Limits of Gateway Endpoint

* You cannot use a prefix list ID in an outbound rule in a network ACL to allow or deny outbound traffic to the service specified in an endpoint. If your network ACL rules restrict traffic, you must specify the CIDR block (IP address range) for the service instead. 
* Endpoints are supported within the same Region only. 
* Endpoints support IPv4 traffic only.
* Cannot transfer an endpoint from one VPC to another,
* Endpoint connections cannot be extended out of a VPC. 
* Must enable DNS resolution in your VPC, or if you're using your own DNS server, ensure that DNS requests to the required service (such as Amazon S3) are resolved correctly to the IP addresses maintained by AWS. 


### Endpoint Service

#### Steps to the connections

* Create a Network Load Balancer for your application in your VPC and configure it for each subnet (Availability Zone) in which the service should be available. 
* Create a VPC endpoint service configuration and specify your Network Load Balancer.
* Grant permissions to specific service consumers (AWS accounts, IAM users, and IAM roles) to create a connection to your endpoint service.
* A service consumer that has been granted permissions creates an interface endpoint to your service, optionally in each Availability Zone in which you configured your service.
* To activate the connection, accept the interface endpoint connection request. By default, connection requests must be manually accepted. However, you can configure the acceptance settings for your endpoint service so that any connection requests are automatically accepted.
* To help achieve high availability for service consumers that use zonal DNS hostnames to access the service, you can enable cross-zone load balancing. Cross-zone load balancing enables the load balancer to distribute traffic across the registered targets in all enabled Availability Zones.

#### Endpoint Service DNS Names

AWS generates endpoint-specific DNS hostnames that you can use to communicate with the service. These names include the VPC endpoint ID, the Availability Zone name and Region Name, for example, vpce-1234-abcdev-us-east-1.vpce-svc-123345.us-east-1.vpce.amazonaws.com. By default, your consumers access the service with that DNS name and usually need to modify the application configuration. 


#### Endpoint Service Limitations

* An endpoint service supports IPv4 traffic over TCP only.
* Service consumers can use the endpoint-specific DNS hostnames to access the endpoint service, or the private DNS name.
* If an endpoint service is associated with multiple Network Load Balancers, then for a specific Availability Zone, an interface endpoint establishes a connection with *one load balancer only.
* For the endpoint service, the associated Network Load Balancer can support 55,000 simultaneous connections or about 55,000 connections per minute to each unique *target (IP address and port). 
* Availability Zones in your account might not map to the same locations as Availability Zones in another account. 
* Review the service-specific limits for your endpoint service.



#### VPC Endpoint Policies

A VPC endpoint policy is an IAM resource policy that you attach to an endpoint when you create or modify the endpoint. If you do not attach a policy when you create an endpoint, AWS attaches a default policy for you that allows full access to the service. 



### VPC Peering

A VPC peering connection is a networking connection between two VPCs that enables you to route traffic between them privately. Instances in either VPC can communicate with each other as if they are within the same network. You can create a VPC peering connection between your own VPCs, with a VPC in another AWS account, or with a VPC in a different AWS Region.

AWS uses the existing infrastructure of a VPC to create a VPC peering connection; it is neither a gateway nor an AWS Site-to-Site VPN connection, and does not rely on a separate piece of physical hardware. There is no single point of failure for communication or a bandwidth bottleneck.

### VPN Connections

VPN connectivity option |   Description 
  -----  |:-----------
AWS Site-to-Site VPN |You can create an IPsec VPN connection between your VPC and your remote network. On the AWS side of the Site-to-Site VPN connection, a virtual private gateway provides two VPN endpoints (tunnels) for automatic failover. You configure your customer gateway on the remote side of the Site-to-Site VPN connection. For more information, see the AWS Site-to-Site VPN User Guide, and the AWS Site-to-Site VPN Network Administrator Guide.
AWS Client VPN 	| AWS Client VPN is a managed client-based VPN service that enables you to securely access your AWS resources in your on-premises network. With AWS Client VPN, you configure an endpoint to which your users can connect to establish a secure TLS VPN session. This enables clients to access resources in AWS or an on-premises from any location using an OpenVPN-based VPN client. For more information, see the AWS Client VPN User Guide.
AWS VPN CloudHub 	| If you have more than one remote network (for example, multiple branch offices), you can create multiple AWS Site-to-Site VPN connections via your virtual private gateway to enable communication between these networks. For more information, see Providing Secure Communication Between Sites Using VPN CloudHub in the AWS Site-to-Site VPN User Guide.
Third party software |  VPN appliance 	You can create a VPN connection to your remote network by using an Amazon EC2 instance in your VPC that's running a third party software VPN appliance. AWS does not provide or maintain third party software VPN appliances; however, you can choose from a range of products provided by partners and open source communities. Find third party software VPN appliances on the AWS Marketplace. <file_sep>+++
date = "2016-12-07T16:56:21+11:00"
title = "Vue 2 CRM Project"

description="Vue2Crm is a reusable Vue.js CRM starter project for real-world business based on Vue 2 PWA template"
+++

## Summary

**Vue2Crm** is a reusable Vue.js CRM starter project for real-world business based on Vue 2 PWA template with Vuetify.  

The goal of this project is to create a reusable project for real-world business. To achieve this target, we need a solution which includes authentication process, restful API feature and simple but elegant UI design. 


## __Features__

* This starter project is built-in with Vue 2 PWA from scratch.
* The whole UI is built on the Vuetify
* It includes Vuex, Axios as well.
* It uses Json-Server as fake Restful API. (You can simple replace it with your own API)

## Structure of Vue2Crm

``` ini
path\to\vue2crm
+---build              <-// webpack files
+---config             <-// configuration of dev or prod environment
+---db                 <-// json files for json-server
|   +---db.json        <-// dummy db
|   \---routes.json    <-// configure fake restful api
+---screenshots
+---src                <-// vue components 
|   +---components
|   |   +---404.vue
|   |   +---About.vue
|   |   +---Customers.vue
|   |   +---Customer.vue
|   |   +---Orders.vue
|   |   +---Order.vue
|   |   +---Login.vue
|   |   \---...
|   +---router         <-// vue-router
|   +---utils
|   |   +---auth.js    <-// auth service
|   |   +---backend-api.js  <-// Axios instance 
|   |   +---store.js   <-//  Vuex
|   \---stylus         <-// Customize stylus
+---static              <-// css, fonts, image files
|   +---img
|   \---manifest.json   <-// PWA manifest file
\---test
    +---e2e
    \---unit

```


## Screenshots

> ![Screenshot1](/img/v2crm-screenshot-1.jpg)

> ![Screenshot2](/img/v2crm-screenshot-2.jpg)

> ![Screenshot3](/img/v2crm-screenshot-3.jpg)

> ![Screenshot4](/img/v2crm-screenshot-4.jpg)



## Browse [Repository](https://github.com/harryho/vue2crm.git)



## __Alternatives__

There are two similar projects respectively built on the Angular and React. If you have interests in those technical stacks. You can find those projects below.

* [Angular4 Crm](/projects/angular4-crm).
* [React Redux Crm](/projects/react-crm).
<file_sep>+++
title="Good practice - 3"
description="Good practice advice - Part 3"
weight=22
+++

### Error


* How to stop a program when an error occurs:

```go
if err != nil {
    fmt.Printf("Program stopping with error %v", err)
    os.Exit(1)
}
// OR :
if err != nil {
    panic("ERROR occurred: " + err.Error())
}
```


### Performance best practices and advice

* Use the initializing declaration form := wherever possible (in functions).
* Use bytes if possible instead of strings
* Use slices instead of arrays.
* Use arrays or slices instead of a map where possible (see ref. 15)
* Use for range over a slice if you only need the value and not the index; this is slightly faster than having to do a slice lookup for every element.
* When the array is sparse (containing many 0 or nil-values), using a map can result in lower memory consumption.
* Specify an initial capacity for maps.
* When defining methods: use a pointer to a type (struct) as a receiver.
* Use constants or flags to extract constant values from the code.
* Use caching whenever possible when large amounts of memory are being allocated.
* Use template caching 



<file_sep>+++
title = "Struct & Interface"
description="Golang Introduction: Struct, Method, Interface & Reflection"
weight=3
+++


### Struct

> Go supports user-defined or custom types in the form of alias types or structs. A struct tries to represent a real-world entity with its properties. Structs are composite types, to use when you want to define a type which consist of a number of properties, each having their own type and value, grouping pieces of data together.

#### Struct with tags

* A field in a struct can, apart from a name and a type, also optionally have a tag: this is a string attached to the field, which could be documentation or some other important label. The tag-content cannot be used in normal programming, only the package reflect can access it.

* Sample

```go
package main
import (
"fmt"
"reflect"
)

type TagType struct { //tags
    field1 bool   "An important answer"
    field2 string "The name of the thing"
    field3 int    "How much there are"
}

func main() {
    tt := TagType{true, "Barak Obama", 1}
    for i := 0; i < 3; i++ {
        refTag(tt, i)
    }
}
func refTag(tt TagType, ix int) {
    ttType := reflect.TypeOf(tt)
    ixField := ttType.Field(ix)
    fmt.Printf("%v\n", ixField.Tag)
}

```

#### Anonymous fields and embedded structs

* It can only have one anonymous field of each data type in a struct.
* The inner struct is simply inserted or “embedded” into the outer. This simple ‘inheritance’ mechanism provides a way to derive some or all of your implementation from another type or types.

```go
type innerS struct {
    in1 int
    in2 int
}
type outerS struct {
    b int
    c float32
    int // anonymous field
    innerS // anonymous field
}
func Func(){
    outer2 := outerS{6, 7.5, 60, innerS{5, 10}}
    fmt.Println("output : ", outer2) // output: {6 7.5 60 {5 10}}
}

```

####  Conflicting names

* The rules when there are two fields with the same name

    *  An outer name hides an inner name. This provides a way to override a field or method.

    * If the same name appears twice at the same level, it is an error if the name is used by the program


### Method

* method is a function that acts on variable of a certain type, called the receiver

* The receiver type can be (almost) anything, not only a struct type: any type can have methods, even a function type or alias types for int, bool, string or array. The receiver also cannot be an interface type.

* sample 

```go
type List []int
func (l List) Len() int { return len(l) }
func (l *List) Append(val int) { *l = append(*l, val) }
```


* Receiver is most often a pointer to the receiver_type for performance reasons. 

#### Embedded methods

* There are basically 2 ways for doing this:
    
    * __Aggregation__ (or composition): include a named field of the type of the wanted functionality 
    
    * __Embedding__: Embed (anonymously) the type of the wanted functionality, like demonstrated


* The aggregated type requires a method to return the pointer. Mostly the method will looks like the type name, as a constructor method 

* The embedded type does not need to a pointer

#### Multiple inheritance 

```go

type Phone struct {}

func (*Phone) Call() { 
    return "Ring Ring !!! "
}

type Camera struct {}

func (*Camera) TakePhoto() {
    return "Take a photo !!!"
}

type CameraPhone struct {
    Phone
    Camera
}

func main () {
    cp := new(CameraPhone)
    fmt.Println( cp.Call())
    fmt.Println( cp.TakePhoto())
}
```


### Interface 

* Go contains the very flexible concept of interfaces, with which a lot of aspects of object-orientation can be made available. Interfaces in Go provide a way to specify the behavior of an object: if something can do this, then it can be used here.

* Sample of interface syntax

```go
type Namer interface {
Method1(param_list) return_type
Method2(param_list) return_type
...
}
```


* Type switch

```go
type Shape interface {
    Area() float32
}
type Square struct {
    side float32
}

type Circle struct {
    radius float32
}

func (sq *Square) Area() float32 {
    return sq.side * sq.side
}

func (c *Circle) Area() float32 {
    return c.radius * c.radius * math.Pi
}

type Rectangle struct {
    length, width float32
}

func (r Rectangle) Area() float32 {
    return r.length * r.width
}

func main() {
    shapes := []Shape{Rectangle{5, 3}, &Square{5}, &Circle{5}}
    fmt.Println("Looping through shapes for area ...")
    for n, _ := range shapes {
        fmt.Println("Shape details: ", shapes[n])
        fmt.Println("Area of this shape is: ", shapes[n].Area())
    }

    var shape Shape
    shape = &Square{4}

    switch t := shape.(type) {
    case *Square:
        fmt.Printf("Type Square %T with value %v\n", t, t)
    case *Circle:
        fmt.Printf("Type Circle %T with value %v\n", t, t)
    case *Rectangle:
        fmt.Printf("Type Rectangle %T with value %v\n", t, t)
    case nil:
        fmt.Println("nil value: nothing to check?")
    default:
        fmt.Printf("Unexpected type %T", t)
    }
}

// Looping through shapes for area ...
// Shape details:  {5 3}
// Area of this shape is:  15
// Shape details:  &{5}
// Area of this shape is:  25
// Shape details:  &{5}
// Area of this shape is:  78.53982
// Type Square *main.Square with value &{4}
```


#### Empty Interface

* The empty or minimal interface has no methods and so doesn’t make any demands at all. So any variable, any type implements it (not only reference types as Object in Java/C#), and any or Any (Sample code below) is really a good name as alias and abbreviation!

```go
type Any interface{}
```

* Type classifier 

```go
func classifier(items ...interface{}) {
    for i, x := range items {
        switch x.(type) {
        case bool:
            fmt.Printf("param #%d is a bool\n", i)
        case float64:
            fmt.Printf("param #%d is a float64\n", i)
        case int, int64:
            fmt.Printf("param #%d is an int\n", i)
        case nil:
            fmt.Printf("param #%d is nil\n", i)
        case string:
            fmt.Printf("param #%d is a string\n", i)
        default:
            fmt.Printf("param #%d’s type is unknown\n", i)
        }
    }
}
```

#### Interface to interface

* An interface value can also be assigned to another interface value, as long as the underlying value implements the necessary methods.

* This conversion is checked at runtime, and when it fails a runtime error occurs: this is one of the dynamic aspects of Go, comparable to dynamic languages like Ruby and Python.

### Reflection

#### Methods and types in reflect

* Reflection in computing is the ability of a program to examine its own structure, particularly through the types; it’s a form of metaprogramming.

* Two simple functions, reflect.TypeOf and reflect.ValueOf, retrieve Type and Value pieces out of any value.


#### Modify a value through reflection

* Pass the address instead of copy of value
* Use the Elem() function work on it which indirects through the pointer

```go
func modifyValByReflect() {
    var x float64 = 3.4
    v := reflect.ValueOf(x)                      // Pass value
    fmt.Println("Settability of v:", v.CanSet()) // false
    v = reflect.ValueOf(&x)                      // Note: take the address of x.
    fmt.Println("type of v:", v.Type())          // float64
    fmt.Println("Settability of v:", v.CanSet()) // false
    v = v.Elem()
    fmt.Println("The Elem of v is: ", v)         // <float64 Value>
    fmt.Println("Settability of v:", v.CanSet()) // true
    v.SetFloat(3.1415)                           // this works!
    fmt.Println(v.Interface())
    fmt.Println(v) // <float64 Value>
}
```

#### Dynamic typing 

* A refactoring pattern that is very useful is extracting interfaces, reducing thereby the number of types and methods needed, without having the need to manage a whole class-hierarchy as in more traditional class-based OO-languages.

* Go is the only one which combines interface values, static type checking (does a type implement the interface?), dynamic runtime conversion and no requirement for explicitly declaring that a type satisfies an interface. This property also allows interfaces to be defined and used without having to modify existing code.

### OO of Go

* Encapsulation (data hiding): in contrast to other OO languages where there are 4 or more access-levels, Go simplifies this to only 2:
    
    * package scope: ‘object’ is only known in its own package, how? it starts with a lowercase letter
    
    * exported: ‘object’ is visible outside of its package, how? it starts with an uppercase letter A type can only have methods defined in its own package.

* Inheritance: how? composition: embedding of 1 (or more) type(s) with the desired behavior (fields and methods); multiple inheritance is possible through embedding multiple types

* Polymorphism: how? interfaces: a variable of a type can be assigned to a variable of any interface it implements. Types and interfaces are loosely coupled, again multiple inheritance is possible through implementing multiple interfaces. Go’s interfaces aren’t a variant on Java or C# interfaces, they’re much more: they are independent and are key to large-scale programming and adaptable, evolutionary design.

<file_sep>+++
title = "Hello World"
draft=false
+++

<!-- >*Hello World* is the classic starter project for any programmer.  -->

>*After I tried [go hugo](https://gohugo.io/) a few years ago, I found it is a no-brainer WordPress alternative tool to manage my site with github. Honestly, I am not a passionate blogger. My blog is just a personal notes or gallery of projects and experiences, which comes from work, hobby and individual projects. Github is definitely the developer's favor instead of mainstream blogger, so I won't suggest everyone to dump your WordPress. Comparing to markdown and command line, WordPress's UI is much more user friendly to most people. If you want to give a shot on your own, you can check out my note - [Create a blog on GitHub Pages](/blogs/create-a-blog-on-github), it shows you how I did it on my machine and make it available on your github page.* 

>*The `Hello World` home page comes from my old site. I am too lazy to write a new one. It has been updated for few times by adding some new programming languages and it is not written for developer.*


```bash
    Dutch : "Hello wereld",
    English : "Hello world",
    French : "Bonjour monde",
    German : "Hallo Welt",
    Greek : "γειά σου κόσμος",
    Italian : "Ciao mondo",
    Japanese : "こんにちは世界",
    Korean : "여보세요 세계",
    Mandarin : "你好世界",
    Portuguese : "Olá mundo",
    Russian : "Здравствулте мир",
    Spanish : "Hola mundo"
```

The different languages above express the same meaning "Hello World". Basically most software engineers start writing "Hello World" application from the very beginning with different programming language. I hope after you read through this page, you will find programming is not rock science and everyone can code for fun. 

As I mentioned before, here is for beginner to have fun to code sth, so I just pick some language which is easy to start or popular you might already heard or convenient for people to try it on your own. The languages below are simply ordered by alphabet.


###  Assumptions

* You have a proper computer instead of a tablet or ipad. Actually you can write some code to create an app directly on your smart phone or tablet after installing some development apps. *AIDE*, *DroidScript* and *QPython* are such applications you can try if you want to play around. 

* If you use Mac, the code for Linux should work on Mac as well. You know how to start a terminal on Mac or a command prompt on Windows. Finally, you are ready to get your hands dirty. 


### Online Programming Editor

I list some online programming editor and compiler here, then you can simply try without installing any framework or tool on your PC or laptop.

* [repl.it](https://repl.it/languages )
* [tutorialspoint codingground](https://www.tutorialspoint.com/codingground.htm)

###  Bash/Batch script

`Bash, aka shell, is built-in script on Unix/Linux-like operating system. Bash script file is end with `sh` as extension. Batch is built-in script on Windows operating system. Bash script is known as one of Unix shell scripts. The other shell scrips include ksh, csh, zsh, etc. Bash is one of most important and  powerful tool for system admin. 

`Batch` script file is end with `bat` as extension. On Windows there is another file end with `cmd`, it works the same as batch file. From November 2006, Microsoft create a new powerful language `PowerShell`, which is similar to Unix shells. Basically `PowerShell` has replaced `Batch` as first option for system admin.

* Unix/Linux 

    * Create a script file `hello.sh` with vi or nano

    ```bash
    #!/bin/bash
    var="Hello World!"
    echo $var
    ```

    * Change mode `chmod 755 hello.sh`
    * Run the script `./hello.sh`

* Batch/Cmd (Windows) 

    * Create a script file hello.bat or hello.cmd with any editor

    ```batch
    @echo off
    set var='Hello World!'
    echo.%var%
    ```

    * Run the script `hello.bat` or `hello.cmd`

###  C/C++

The C programming language was originally developed as a language to replace assembler in systems programming. It was very successful, making system code portable and easier to write and read. So Basically the kernel of most operating systems,  Windows, Mac, Linux are coded in C. 

Today C is one of the most used programming languages. Since C was designed to replace assembler language, and that in several important ways, it retains a very low level view of the machine. The C++ programming language was designed as a higher level version of C, providing support for object-oriented programming. It gives developer more power to handle the problem of real world. 

* Create a program file `hello.c ` or `hello.cpp` with vi or any editor

    ```cpp
    #include<stdio.h>

    int main( ) 
    {
        char var[] = "World";   
        printf( "Hello %s \n Press any key to exit.", var );
        char key = getchar();
        return 0;
    }
    ```

* Unix/Linux includes `gcc` by default. You just need to compile and run the console app. For Windows, you might need install another tool [cygwin](https://cygwin.com/install.html) or [MinGW](http://www.mingw.org/wiki/MinGW_for_First_Time_Users_HOWTO)

    ```
    g++ hello.c -o hello
    ./hello
    ```

###  C\#

C# is a multi-paradigm programming language encompassing strong typing, imperative, declarative, functional, generic, object-oriented (class-based), and component-oriented programming disciplines. In January 1999, <NAME> formed a team to build a new language at the time called Cool, which stood for "C-like Object Oriented Language". By the time the .NET project was publicly announced at the July 2000,Microsoft the language had been renamed C#. 

* Download and install .net framework from Microsoft website
* Create a program file helloworld.cs

    ```cs
    public class Hello
    {
        public static void Main()
        {
            System.Console.WriteLine("Hello, World!");
        }
    }
    ```

* Compile with .net framework command. 
* Run helloworld.exe file 

    ```bash
    C:\Windows\Microsoft.NET\Framework\v3.5\csc.exe helloworld.cs
    helloworld.exe
    ```

###  Go

Go ,as known as golang, is a free and open source programming language created at Google. It is a compiled, statically typed language in the tradition of Algol and C, with garbage collection, expressive, concise, clean, and efficient. Its concurrency mechanisms make it easy to write programs that get the most out of multicore and networked machines, while its novel type system enables flexible and modular program construction. 

* Download Go binary from www.golang.org
* Follow the instruction to [install](https://golang.org/doc/install) golang on your computer
* Create a program file hello.go

    ```go
    package main

    import "fmt"
    func main() {
        fmt.Println("hello world")
    }
    ```

* Compile & run `go run hello.go`
  
###  Java

Java is a general-purpose computer programming language that is concurrent, object-oriented, and specifically designed to have as few implementation dependencies as possible. It is intended to let application developers "write once, run anywhere", meaning that compiled Java code can run on all platforms with JVM without the need for recompilation.

* Download and install [Java JDK 8](http://www.oracle.com/technetwork/java/javase/downloads/index.html)
      * For any Ubuntu 12 or higher version I recommand you follow this [instruction](http://www.webupd8.org/2012/09/install-oracle-java-8-in-ubuntu-via-ppa.html). It is pretty simple. 
      * For CentOS 6 or higher version, please follow this [instruction](https://wiki.centos.org/HowTos/JavaRuntimeEnvironment)
      * For Windows please make sure you click the [JDK Download](http://www.oracle.com/technetwork/java/javase/downloads/index.html) button. The installer file end with `exe` is best option for beginner, instead of the zip file, because you don't need to setup **PATH** system environment by yourself.  

* Create a program file HelloWorld.java

    ```java
    public class HelloWorld {
        public static void main(String[] args) {
            System.out.println("Hello, World");
        }
    }
    ```

* Compile & run 

    ```
    javac HelloWorld.java
    java HelloWorld
    ```
  
###  Javascript

JavaScript, not to be confused with Java, was created in 10 days in May 1995 by <NAME>, then working at Netscape and now of Mozilla. The original name of this language was Mocha, in September of 1995 it was changed to LiveScript, then in December of the same year, the name JavaScript was adopted, because of very popular Java around then. 

JavaScript is the programming language of the web. It's one of the most popular and in demand skills in today's job market for good reason. As a web developer, it is essential that you have a solid understanding of this versatile language.

* The sample will be presented as web page.
* Create a program file HelloWorld.html

    ```html
    <!doctype html>
    <html>
        <head>
            <script>
            function helloWorld() {
                document.write("Hello World");
            }
            helloWorld();
            </script>
        </head>
    </html>
    ```
    
* Open file HelloWorld.html with your favorite browser. 


###  PHP

PHP as it's known today is actually the successor to a product named PHP/FI. Created in 1994 by <NAME>, the very first incarnation of PHP was a simple set of Common Gateway Interface (CGI) binaries written in the C programming language. Originally used for tracking visits to his online resume, he named the suite of scripts "Personal Home Page Tools," more frequently referenced as "PHP Tools." Over time, more functionality was desired, and Rasmus rewrote PHP Tools, producing a much larger and richer implementation. 

* Download and install 

**Linux**
Please find the install command [here](/blog/ubuntu-server-14/#install-php-compser)

**Windows**    
    *  Download the file [PHP 5.x here](http://php.net/downloads.php) 
    *  Pick the Thread safe zip file, download extra it to \path\to\php_folder. 
    *  Update the PATH environment variable with your php directory

* Create a python script hello.py with any editor

    ```php
    <?php
        echo "Hello World!" 
   ?>
    ```
    
* Run the script `php hello.php` 

###  Python

Python is currently one of the most popular dynamic programming languages, along with Perl, Tcl, PHP, and newcomer Ruby. Although it is often viewed as a "scripting" language, it is really a general purpose programming language along the lines of Lisp or Smalltalk (as are the others, by the way). Today, Python is used for everything from throw-away scripts to large scalable web servers that provide uninterrupted service 24x7. It is used for GUI and database programming, client- and server-side web programming, and application testing. It is used by scientists writing applications for the world's fastest supercomputers and by children first learning to program.

* Download and install [python 3.x](https://www.python.org/downloads/)
* Create a python script hello.py with any editor

    ```python
    print "Hello World!"
    ```
    
* Run the script 
        
        python hello.py 


###  Rust 

Rust is a systems programming language that runs blazingly fast, prevents segfaults, and guarantees thread safety. Rust programming language is fundamentally about empowerment: no matter what kind of code you are writing now, Rust empowers you to reach farther, to program with confidence in a wider variety of domains than you did before.

* Download and install rust  [rust 1.x](https://www.rust-lang.org/en-US/install.html)

* Create a project 

    `cargo new helloworld`

* Copy the content below into the rust file main.rs with any editor

    ```rust
    fn main() {
        println!("Hello World !")
    }
    ```

* Run with rust app with cargo

        cargo run




  

<file_sep>#!/bin/bash
echo $1

PROFILE=$1 

domains=$(aws route53 list-hosted-zones  \
--profile ${PROFILE} \
--out text | awk '{print $4}'  ) 

# echo $domains
for domain in "${domains[@]}"; do
        printf "${domain/"\.\s"/"\\t\\n"}"

done;

<file_sep>+++
date = "2013-09-06T16:43:12+11:00"
title = "Do you have potential to be a programmer?"
hidden=true
draft=true
+++

<!--### Do you have potential to be a programmer?-->

>*For many novices or beginners of software engineering or computer programming, they always question themselves like this "Do I have potential to be a programmer?" or "Should I choose programmer as my career?" or "Is it programmer right for me?". I am pretty sure you can find tons of answers or advices online, but there is a simple answer for such question for any career. If you love what you do, then it is right for you. It brings another question, how do I know if I love it or not. Not everyone has chance to try different jobs before they are qualified.*

>*There is a test created for this purpose. It is designed for people who have not enough programming skills but they want to know if they have such potential to be a programmer.* 

>*The test needs you to create a few "Hello World" programs as the samples below with different programming languages. The programming languages I pick can run on Windows, Mac or Linux/Unix. You should try to complete the test on your own. There is no time limit.*

>*How much time you take to complete these programs or how many programs you can complete is not the purpose of this test, but you should try your best to complete as much as possible. I have to say it is not an easy task for most novices or beginners. Even you can't complete all programs, it doesn't mean you can't be a great programmer. I believe everyone can learn or do anything if they love or enjoy it.*

__So the test is to check if you really enjoy the process or problem-solving when you face to such challenges. If you always can learn something after you solve problem or you keep improving your skills during the test, it means you really love it and you can consider programmer as one of your future careers.__

__If you complete the test within 2 hours to one day from scratch, it means you already have capacity to be a programmer. It means the test is not designed for you.__


### Assumptions

* You have a proper computer instead of a tablet or ipad. 

* You know how to download and install softwares on your computer.

* You know how to organize your folders and files.

* If you use Mac, the code for Linux should work on Mac as well. You know how to start a terminal on Mac or a command prompt on Windows. Finally, you are ready to get your hands dirty. 


### Bash/Batch script

`Bash` is built-in script on Unix/Linux-like operating system. Bash script file is end with `sh` as extension. Batch is built-in script on Windows operating system. Bash script is known as one of Unix shell scripts. The other shell scrips include ksh, csh, zsh, etc. Bash is one of most important and  powerful tool for system admin. 

`Batch` script file is end with `bat` as extension. On Windows there is another file end with `cmd`, it works the same as batch file. From November 2006, Microsoft create a new powerful language `PowerShell`, which is similar to Unix shells. Basically `PowerShell` has replaced `Batch` as first option for system admin.

* Unix/Linux 

    * Create a script file `hello.sh` with vi or nano

    ```bash
    #!/bin/bash
    var="Hello World!"
    echo $var
    ```

    * Change mode `chmod 755 hello.sh`
    * Run the script `./hello.sh`

* Batch/Cmd (Windows) 

    * Create a script file hello.bat or hello.cmd with notepad

    ```dos
    @echo off
    set var='Hello World!'
    echo.%var%
    ```

    * Run the script `hello.bat` or `hello.cmd`

### C/C++

The C programming language was originally developed as a language to replace assembler in systems programming. It was very successful, making system code portable and easier to write and read. So Basically the kernel of most operating systems,  Windows, Mac, Linux are coded in C. 

Today C is one of the most used programming languages. Since C was designed to replace assembler language, and that in several important ways, it retains a very low level view of the machine. The C++ programming language was designed as a higher level version of C, providing support for object-oriented programming. It gives developer more power to handle the problem of real world. 

## Windows

* Download and install C/C++ development and compiler. [Instructions](http://www.cprogramming.com/code_blocks/)
* Create a program file `hello.c ` or `hello.cpp` with codeblocks, vi or notepad

    ```cpp
    #include<stdio.h>

    int main( ) 
    {
        char var[] = "World";   
        printf( "Hello %s \n Press any key to exit.", var );
        char key = getchar();
        return 0;
    }
    ```

## Unix/Linux 

Unix/Linux includes `gcc` by default. You just need to compile and run the console app. For Windows, you might need install another tool [cygwin](https://cygwin.com/install.html) or [MinGW](http://www.mingw.org/wiki/MinGW_for_First_Time_Users_HOWTO)

    ```
    g++ hello.c -o hello
    ./hello
    ```

### C\# ##

C# is a multi-paradigm programming language encompassing strong typing, imperative, declarative, functional, generic, object-oriented (class-based), and component-oriented programming disciplines. In January 1999, <NAME> formed a team to build a new language at the time called Cool, which stood for "C-like Object Oriented Language". By the time the .NET project was publicly announced at the July 2000,Microsoft the language had been renamed C#. 

## Windows
* Download and install .net framework from Microsoft website
* Create a program file helloworld.cs

    ```cs
    public class Hello
    {
        public static void Main()
        {
            System.Console.WriteLine("Hello, World!");
        }
    }
    ```

* Compile with .net framework command. 
* Run helloworld.exe file 

    ```bash
    C:\Windows\Microsoft.NET\Framework\v3.5\csc.exe helloworld.cs
    helloworld.exe
    ```

## Linux/Mac
### Mono
* Download and install mono framework and IDE for Linux or Mac from [here](http://www.mono-project.com/download/)
* Create a program file helloworld.cs as above in the IDE and run

### .Net Core
* Download and install .Net Core framework for Linux or mac from [here](https://www.microsoft.com/net/download/core)
* Follow the instructions to create a "Hello World" app




### Go

Go ,as known as golang, is a free and open source programming language created at Google. It is a compiled, statically typed language in the tradition of Algol and C, with garbage collection, expressive, concise, clean, and efficient. Its concurrency mechanisms make it easy to write programs that get the most out of multicore and networked machines, while its novel type system enables flexible and modular program construction. 

* Download Go binary from www.golang.org
* Follow the instruction to [install](https://golang.org/doc/install) golang on your computer
* Create a program file hello.go

    ```go
    package main

    import "fmt"
    func main() {
        fmt.Println("hello world")
    }
    ```

* Compile & run `go run hello.go`
  
### Java

Java is a general-purpose computer programming language that is concurrent, object-oriented, and specifically designed to have as few implementation dependencies as possible. It is intended to let application developers "write once, run anywhere", meaning that compiled Java code can run on all platforms with JVM without the need for recompilation.

* Download and install [Java JDK 8](http://www.oracle.com/technetwork/java/javase/downloads/index.html)
      * For any Ubuntu 12 or higher version I recommand you follow this [instruction](http://www.webupd8.org/2012/09/install-oracle-java-8-in-ubuntu-via-ppa.html). It is pretty simple. 
      * For CentOS 6 or higher version, please follow this [instruction](https://wiki.centos.org/HowTos/JavaRuntimeEnvironment)
      * For Windows please make sure you click the [JDK Download](http://www.oracle.com/technetwork/java/javase/downloads/index.html) button. The installer file end with `exe` is best option for beginner, instead of the zip file, because you don't need to setup **PATH** system environment by yourself.  

* Create a program file HelloWorld.java

    ```java
    public class HelloWorld {
        public static void main(String[] args) {
            System.out.println("Hello, World");
        }
    }
    ```

* Compile & run 

    ```
    javac HelloWorld.java
    java HelloWorld
    ```
  
### JavaScript

JavaScript, not to be confused with Java, was created in 10 days in May 1995 by <NAME>, then working at Netscape and now of Mozilla. The original name of this language was Mocha, in September of 1995 it was changed to LiveScript, then in December of the same year, the name JavaScript was adopted, because of very popular Java around then. 

JavaScript is the programming language of the web. It's one of the most popular and in demand skills in today's job market for good reason. As a web developer, it is essential that you have a solid understanding of this versatile language.

* The sample will be presented as web page.
* Create a program file HelloWorld.html

    ```html
    <!doctype html>
    <html>
        <head>
            <script>
            function helloWorld() {
                document.write("Hello World");
            }
            helloWorld();
            </script>
        </head>
    </html>
    ```
    
* Open file HelloWorld.html with your browser. 


### PHP

PHP as it's known today is actually the successor to a product named PHP/FI. Created in 1994 by <NAME>, the very first incarnation of PHP was a simple set of Common Gateway Interface (CGI) binaries written in the C programming language. Originally used for tracking visits to his online resume, he named the suite of scripts "Personal Home Page Tools," more frequently referenced as "PHP Tools." Over time, more functionality was desired, and Rasmus rewrote PHP Tools, producing a much larger and richer implementation. 

* Download and install 

**Linux**
Please find the install command [here](/os/ubuntu-server-14/#install-php-compser)

**Windows**    
    *  Download the file [PHP 5.x here](http://php.net/downloads.php) 
    *  Pick the Thread safe zip file, download extra it to \path\to\php_folder. 
    *  Update the PATH environment variable with your php directory

* Create a python script hello.py with notepad

    ```php
    <?php
        echo "Hello World!" 
   ?>
    ```
    
* Run the script `php hello.php` 

### Python

Python is currently one of the most popular dynamic programming languages, along with Perl, Tcl, PHP, and newcomer Ruby. Although it is often viewed as a "scripting" language, it is really a general purpose programming language along the lines of Lisp or Smalltalk (as are the others, by the way). Today, Python is used for everything from throw-away scripts to large scalable web servers that provide uninterrupted service 24x7. It is used for GUI and database programming, client- and server-side web programming, and application testing. It is used by scientists writing applications for the world's fastest supercomputers and by children first learning to program.

* Download and install [python 3.x](https://www.python.org/downloads/)
* Create a python script hello.py with any file editor

    ```python
    print "Hello World!"
    ```
    
* Run the script `python hello.py` 

### Ruby 

Ruby is a dynamic, reflective, object-oriented, general-purpose programming language. Ruby was influenced by Perl, Smalltalk, Eiffel, Ada, and Lisp. It supports multiple programming paradigms, including functional, object-oriented, and imperative. It also has a dynamic type system and automatic memory management.

* Download and install proper [Ruby](https://www.ruby-lang.org/en/documentation/installation/)
* Create a ruby program hello.rb with with any file editor

    ```ruby
    print "Hello World!"
    ```
* Run the program `ruby hello.rb`    



  

<file_sep>// var a = "abc"
// console.log(a);

// this.a = "123";
// console.log(a);

// console.log(this);

// foo = 'abc';
// alert(foo); // abc

// this.foo = 'def';
// alert(foo); //

// console.log(  " 1==true : ", 1==true)
// console.log(  " ''==true : ", ''==true)
// console.log(  " '1'==true : ", '1'==true)
// console.log(  " \"\"==true : ", "1"==true)
// console.log(  " {}==true : ", [{}]==true)
// console.log(  " []==true : ", ['1']==true)

// console.log(  " 0==false : ", 0==false)
// console.log(  " -1==false : ", -1==false)
// console.log(  " ''==false : ", ''==false)
// console.log(  " '1'==false : ", '1'==false)
// console.log(  " \"\"==false : ", ""==false)
// console.log(  " {}==false : ", {}==false)
// console.log(  " []==false : ", []==false)

// console.log(  " null==false : ", null==false)
// console.log(  " undefined==false : ", undefined==false)
// console.log(  " undefined==null : ", null==undefined)

// var obj = {
//   _p: '_f',
//   _f: function() {
//     console.log(' f -> ', this);
//     // console.log(this.__proto__.constructor.name);
//     _ff = {
//       //   console.log(' ff ', this);
//       _p: '_ff',
//       _fff: function() {
//         this.p = '123';
//         console.log(' fff -> ', this);
//         // console.log(this.__proto__.constructor.name);
//       }
//     };
//     _ff._fff();
//   }
// };

// obj._f();

// console.log(' -------------------------------------- ');

// var obj2 = {
//   _p: 123,
//   _f: () => {
//     console.log(' f -> ', this);
//     // console.log(this.__proto__.constructor.name);
//     _ff = {
//       _p: '_ff',
//       _fff: () => {
//         this.p = '123';
//         console.log(' fff -> ', this);

//         //   console.log(this.__proto__.constructor.name);
//       }
//     };
//     _ff._fff();
//   }
// };

// obj2._f();

//////////////////////////////////////////

// var obj3 = {
//   p: 'obj3',
//   toBeCalled: function() {
//     console.log(' this is toBeCalled ', this.p);
//   },
//   toBind: function(obj) {
//     obj.toBeCalled();
//     // console.log(' this is toBind ', obj);
//   }
// };

// var testBind = obj3.toBind;
// testBind(obj3);

// var obj4 = {
//   p: 'obj4',
//   toBeCalled: () => {
//     console.log(' this is toBeCalled ', this.p);
//   },
//   toBind: obj => {
//     obj.toBeCalled();
//     // console.log(' this is toBind ', obj);
//   }
// };

// var testBind2 = obj4.toBind;
// testBind2(obj4);



<file_sep>+++
title = "PowerShell 7"
description = "Empower Windows with modern PowerShell"
+++

__This article will show you how to empower your Windows OS with modern PowerShell & other convenient tools.__

## PowerShell

> PowerShell is a cross-platform task automation solution made up of a command-line shell, a scripting language, and a configuration management framework. PowerShell runs on Windows, Linux, and macOS.

### PowerShell 7

#### Differences between PS 5 and PS 7

Windows PowerShell 5.1 is built on top of the .NET Framework v4.5. With the release of PowerShell 6.0, PowerShell became an open source project built on .NET Core 2.0. PowerShell 7.0 is built on .NET Core 3.1. And, with the release of PowerShell 7.2, PowerShell will be built on .NET 6.0. Moving from the .NET Framework to .NET Core allowed PowerShell to become a cross-platform solution. PowerShell runs on Windows, macOS, and Linux.

There are few differences in the PowerShell language between Windows PowerShell and PowerShell. The differences are most notable in the availability and behavior of PowerShell cmdlets between Windows and non-Windows platforms and the changes that stem from the differences between the .NET Framework and .NET Core.

#### Installation

Install latest PowerShell (version 7.x) from Windows Store



### Windows Terminal

> The Windows Terminal is a modern, fast, efficient, powerful, and productive terminal application for users of command-line tools and shells like Command Prompt, PowerShell, and WSL. Its main features include multiple tabs, panes, Unicode and UTF-8 character support, a GPU accelerated text rendering engine, and custom themes, styles, and configurations.

#### Install Nerd font

- [Nerd Font](https://www.nerdfonts.com/font-downloads)
- Download Meslo Nerd Font
- Install font file

#### Installation

This is an open source project and we welcome community participation. To participate please visit https://github.com/microsoft/terminal

#### Update Setting

- Settings > Startups
    - Profile -> PowerShell ( Default is Windows PowerShell )
    - Launch mode -> Maximised

- Update Setting file

    - Hide the proflle Windows PowerShell & move the PowerShell to the top

    ```json
    "list": [
        {
                "guid": "{574e775e-4f2a-5b96-ac1e-a2962a402336}",
                "hidden": false,
                "name": "PowerShell",
                "source": "Windows.Terminal.PowershellCore"
        },
        ...
        {
                "commandline": "powershell.exe",
                "font": 
                {
                    "face": "MesloLGM NF"
                },
                "guid": "{61c54bbd-c2c6-5271-96e7-009a87ff44bf}",
                "hidden": true,
                "name": "Windows PowerShell"
        } 
    ```

    - Create a customzed color scheme

    ```json
       {
            "background": "#001D26",
            "black": "#282C34",
            "blue": "#61AFEF",
            "brightBlack": "#9aabc5",
            "brightBlue": "#61AFEF",
            "brightCyan": "#56B6C2",
            "brightGreen": "#98C379",
            "brightPurple": "#C678DD",
            "brightRed": "#E06C75",
            "brightWhite": "#DCDFE4",
            "brightYellow": "#E5C07B",
            "cursorColor": "#FFFFFF",
            "cyan": "#56B6C2",
            "foreground": "#DCDFE4",
            "green": "#98C379",
            "name": "One Half Dark (mod)",
            "purple": "#C678DD",
            "red": "#E06C75",
            "selectionBackground": "#FFFFFF",
            "white": "#DCDFE4",
            "yellow": "#E5C07B"
        }

    ```

- Settings > Default > Appearence
  - Color scheme : One Half Dark (mod)
  - Font face:  MesloLGM NF
  - Acrylic opacity: 60% 


### Git

```
winget install -e --id Git.Git
```

### Scoop

> A command-line installer for Windows 

#### Install from [Scoop](https://scoop.sh)

```powershell
Invoke-Expression (New-Object System.Net.WebClient).DownloadString('https://get.scoop.sh')

# or shorter
iwr -useb get.scoop.sh | iex
```

#### Other tools

```
scoop install curl sudo jq 
scoop install neovim gcc 
scoop install 7zip bat 
```

### User Profile

- Create user profile folder

```cmd
mkdir ~\.config\powershell
```

- Create a new profile

```
nvim ~\.config\powershell\profile.ps1
```

- Add alias

```powershell

# Alias
Set-Alias vi nvim
Set-Alias ll ls
Set-Alias g git
Set-Alias grep findstr
Set-Alias tig $env:USERPROFILE\app\git\usr\bin\tig.exe
Set-Alias less $env:USERPROFILE\app\git\usr\bin\less.exe
```

- Update built-in profile

```powershell
# Get all profiles
$PROFILE | Get-Member -Type NoteProperty

nvim $PROFILE.CurrentUserCurrentHost
```

- Update $PROFILE.CurrentUserCurrentHost as below

```
. $env:USERPROFILE\.config\powershell\profile.ps1
```


### Oh My Posh

- Installation

  [Please follow latest instruction](https://ohmyposh.dev/docs/migrating)


- Create customized theme file `sample.omp.json`

```
{
  "$schema": "https://raw.githubusercontent.com/JanDeDobbeleer/oh-my-posh/main/themes/schema.json",
  "final_space": false,
  "osc99": true,
  "blocks": [
    {
      "type": "prompt",
      "alignment": "left",
      "segments": [
        {
          "type": "shell",
          "style": "diamond",
          "leading_diamond": "╭─",
          "trailing_diamond": "",
          "foreground": "#ffffff",
          "background": "#0077c2",
          "properties": {
          }
        },
        {
          "type": "root",
          "style": "diamond",
          "leading_diamond": "",
          "trailing_diamond": "",
          "foreground": "#FFFB38",
          "background": "#ef5350",
          "properties": {
            "root_icon": "\uf292",
            "prefix": "<parentBackground>\uE0B0</> "
          }
        },
        {
          "type": "path",
          "style": "powerline",
          "powerline_symbol": "\uE0B0",
          "foreground": "#E4E4E4",
          "background": "#444444",
          "properties": {
            "style": "full",
            "enable_hyperlink": true
          }
        },
        {
          "type": "git",
          "style": "powerline",
          "powerline_symbol": "\uE0B0",
          "foreground": "#011627",
          "background": "#FFFB38",
          "background_templates": [
            "{{ if or (.Working.Changed) (.Staging.Changed) }}#ffeb95{{ end }}",
            "{{ if and (gt .Ahead 0) (gt .Behind 0) }}#c5e478{{ end }}",
            "{{ if gt .Ahead 0 }}#C792EA{{ end }}",
            "{{ if gt .Behind 0 }}#C792EA{{ end }}"
          ],
          "properties": {
            "branch_icon": "\ue725 ",
            "fetch_status": true,
            "fetch_upstream_icon": true,
            "template": "{{ .HEAD }} {{ if .Working.Changed }}{{ .Working.String }}{{ end }}{{ if and (.Working.Changed) (.Staging.Changed) }} |{{ end }}{{ if .Staging.Changed }}<#ef5350> \uF046 {{ .Staging.String }}</>{{ end }}"
          }
        }
      ]
    },
    {
      "type": "prompt",
      "alignment": "right",
      "segments": [
        {
          "type": "node",
          "style": "diamond",
          "leading_diamond": " \uE0B6",
          "trailing_diamond": "\uE0B4",
          "foreground": "#3C873A",
          "background": "#303030",
          "properties": {
            "prefix": "\uE718 ",
            "postfix": "",
            "display_package_manager": true,
            "yarn_icon": " <#348cba></>",
            "npm_icon": " <#cc3a3a></> "
          }
        },
        {
          "type": "time",
          "style": "diamond",
          "invert_powerline": true,
          "leading_diamond": " \uE0B6",
          "trailing_diamond": "\uE0B4",
          "background": "#40c4ff",
          "foreground": "#ffffff",
          "properties": {
            "prefix": " \uf5ef ",
            "postfix": " "
        }
        }
      ]
    },
    {
      "type": "prompt",
      "alignment": "left",
      "newline": true,
      "segments": [
        {
          "type": "text",
          "style": "plain",
          "foreground": "#21c7c7",
          "properties": {
            "prefix": "",
            "postfix": "",
            "text": "╰─"
          }
        },
        {
          "type": "exit",
          "style": "plain",
          "foreground": "#e0f8ff",
          "properties": {
            "prefix": "\u276F",
            "display_exit_code": false,
            "always_enabled": true,
            "error_color": "#ef5350"
          }
        }
      ]
    }
  ]
}

```


- Update user profile

```powershell

# Prompt
Import-Module posh-git
$omp_config = "$env:USERPROFILE\sample.omp.json"
oh-my-posh --init --shell pwsh --config $omp_config | Invoke-Expression
```


### Posh-Git 

```
Install-Module posh-git
```

### Terminal Icons

- Installation

```powershell
 Install-Module -Name Terminal-Icons -Repository PSGallery -Force
```

- Update user profile

```powershell
Import-Module -Name Terminal-Icons
```

### Z

```
Install-Module -Name z --Respository SGallery -Force
```

### PSReadLine

- Installation


```powershell
Install-Module -Name PSReadLine  -AllowPrerelease -Scope CurrentUser -Force -SkipPublisherCheck
```

- Update user profile

```powershell
# PSReadLine
Set-PSReadLineOption -EditMode Emacs
Set-PSReadLineOption -BellStyle None
Set-PSReadLineKeyHandler -Chord 'Ctrl+d' -Function DeleteChar
Set-PSReadLineOption -PredictionSource History
```

### FZF

- Installation

```powershell
scoop install fzf
Install-Module -Name PSFzf -Scope CurrentUser -Force
```

- Update user profile

```powershell
# Fzf
Import-Module PSFzf
Set-PsFzfOption -PSReadlineChordProvider 'Ctrl+f' -PSReadlineChordReverseHistory 'Ctrl+r'
```


<file_sep>+++
title = "Azure: RBAC - 1"
weight = 1
description="Introduction of RBAC"

+++

## RBAC

Azure RBAC is an authorization system built on Azure Resource Manager that provides fine-grained access management of Azure resources.

### What to do with RBAC

* Allow one user to manage virtual machines in a subscription and another user to manage virtual networks
* Allow a DBA group to manage SQL databases in a subscription
* Allow a user to manage all resources in a resource group, such as virtual machines, websites, and subnets
* Allow an application to access all resources in a resource group

### How it works

The way you control access to resources using Azure RBAC is to create role assignments. This is a key concept to understand – it's how permissions are enforced. A role assignment consists of three elements: security principal, role definition, and scope.


#### Security principal

A security principal is an object that represents a user, group, service principal, or managed identity that is requesting access to Azure resources.

* User - An individual who has a profile in Azure Active Directory.

* Group - A set of users created in Azure Active Directory. 

* Service principal - A security identity used by applications or services to access specific Azure resources.

* Managed identity - An identity in Azure Active Directory that is automatically managed by Azure.


#### Role definition

A role definition is a collection of permissions. It's typically just called a role. A role definition lists the operations that can be performed, such as read, write, and delete.

Built-in roles:

* Owner - Has full access to all resources including the right to delegate access to others.
* Contributor - Can create and manage all types of Azure resources but can't grant access to others.
* Reader - Can view existing Azure resources.
* User Access Administrator - Lets you manage user access to Azure resources.


#### Scope

Scope is the set of resources that the access applies to. When you assign a role, you can further limit the actions allowed by defining a scope.


Scopes are structured in a parent-child relationship:management group, subscription, resource group, or resource.

{{<mermaid>}}
graph TB
    MG(Management Group)
    S1(Subscrition)
    S2(Subscrition)
    RG1(Resource Group)
    RG2(Resource Group)
    R1(Resource)
    R2(Resource)
    MG --- S1
    MG --- S2
    S2 --- RG1
    S2 --- RG2
    RG1 --- R1
    RG1 --- R2
{{</mermaid >}}


#### Role assignments

A role assignment is the process of attaching a role definition to a user, group, service principal, or managed identity at a particular scope for the purpose of granting access. Access is granted by creating a role assignment, and access is revoked by removing a role assignment.

{{<mermaid>}}
stateDiagram
    SP:Security Principal
    RA:Role_Assignment
    S:Scope
    RD:Role_Definition
    S --> RA
    SP --> RA
    RD --> RA

{{</mermaid >}}


#### Multiple role assignments

Azure RBAC is an additive model, so your effective permissions are the sum of your role assignments. The sum of the Contributor permissions and the Reader permissions is effectively the Contributor role for the resource group. Therefore, in this case, the Reader role assignment has no impact.

#### Deny assignments

Previously, Azure RBAC was an allow-only model with no deny, but now Azure RBAC supports deny assignments in a limited way. Similar to a role assignment, a deny assignment attaches a set of deny actions to a user, group, service principal, or managed identity at a particular scope for the purpose of denying access.

A role assignment defines a set of actions that are allowed, while a deny assignment defines a set of actions that are not allowed. In other words, deny assignments block users from performing specified actions even if a role assignment grants them access.




<file_sep>+++

title = "String & Representation"
description="String & Representation"
weight=3
+++

## String & Representation

### str()
* `print()` -> `str()` ->  `__str(self)__`
* Fallback to `repr()`. By default, `str()` simply calls `repr()`
* Produces a readable, human-friendly representation of an object
* It is also the string constructor


### repr()

* Exactness is more important than human-friendliness
* Suited for debugging. Unambiguous, precise, include type
* Includes identifying information. 
* Generally best for logging and developers
* The default repr() is not very helpful
* As a rule, you should always write a repr() for your classes
* standard library `reprlib.repr()` is a replacement for `repr()`
    * Example

    ```python
    >>> l = ['a'] * 1000
    >>> import reprlib
    >>> reprlib.repr(l)
    "['a', 'a', 'a', 'a', 'a', 'a', ...]"
    ```

### format

* `"{:f}".format(obj)` -> `__format__(self, f)`
* Fallback to `str()`



### built-in functions

* `ascii()` replaces non-ASCII characters with escape sequences
* `chr()` converts an integer Unicode codepoint to a single character string
* `ord()` converts a single character to its integer Unicode codepoint



## Numeric & scalar types

### int

* unlimited precision signed integer
* bool in an int

    ```python
    >>> False - True
    -1
    >>> False - False
    0
    >>> True - False
    1
    >>> True - True
    0
    ```

### float 

* IEEE-754 double precision (64-bit)
* 53 bits of binary precision
* 15 to 17 bits of decimal precision
* Floating-point numbers are represented in computer hardware as base 2 (binary) fractions.
    
    * 0.001 has value 0/2 + 0/4 + 1/8. These two fractions have identical values, the only real difference being that the first is written in base 10 fractional notation, and the second in base 2
    
    * Unfortunately, most decimal fractions cannot be represented exactly as binary fractions. A consequence is that, in general, the decimal floating-point numbers you enter are only approximated by the binary floating-point numbers actually stored in the machine.

* Example of float data

    ```python

    >>> f=0.9-0.8
    >>> f
    0.09999999999999998
    >>> f=0.2-0.1
    >>> f
    0.1
    >>> f=0.8-0.7
    >>> f
    0.10000000000000009
    >>> float(2**53)
    9007199254740992.0
    >>> float(2**53+1)
    9007199254740992.0
    >>> float(2**53+2)
    9007199254740994.0
    >>> float(2**53+3)
    9007199254740996.0
    >>> float(2**53+4)
    9007199254740996.0

    ```


### decimal

* standard library module decimal containing the class Decimal
* decimal floating point configurable (although finite) precision defaults to 28 digits of decimal precision
* identity is preserved. `x == (x // y) * y + x % y`, so integer division and modulus are consistent
* Example

    ```python
    >>> from decimal import Decimal
    >>> Decimal(0.6)-Decimal(0.5)
    Decimal('0.09999999999999997779553950750')
    >>> Decimal('0.6')-Decimal('0.5')
    Decimal('0.1')
    >>> Decimal(0.2)-Decimal(0.1)
    Decimal('0.1000000000000000055511151231')
    >>> Decimal(0.8)-Decimal(0.7)
    Decimal('0.1000000000000000888178419700')
    >>> Decimal('0.8')-Decimal('0.7')
    Decimal('0.1')

    ## Change the precise 
    >>> import decimal
    >>> decimal.getcontext().prec=4
    >>> Decimal(-7) / Decimal(3)
    Decimal('-2.333')

    ## Python handle different type in different way with % 
    >>> Decimal(-7) % Decimal(3)
    Decimal('-1')  ## 
    >>> Decimal(-7) // Decimal(3)
    Decimal('-2')  ## The next multiple of 3 towards zero is -6
    >>> (-7) // (3) 
    -3  ## The largest multiple of 3 less than -7 is -9
    >>> (-7) % (3)
    2  ## The 
    ```

### fraction

* standard library module fractions containing the class Fraction for rational numbers
    * Denominator cannot be zero. e.g. 2 / 3, 2 is numerator, 3 is denominator.

* Example

    ```python

    >>>from fractions import Fraction
    >>>f = Fraction("2/3")
    >>>f
    Faction(2,3)
    >>> Fraction(0.2)
    Fraction(3602879701896397, 18014398509481984)
    >>> Fraction(0.5)
    Fraction(1, 2)
    >>> Fraction(Decimal('0.3'))
    Fraction(3, 10)
    >>> Fraction(Decimal('0.3')) // Fraction(6, 7)
    0
    >>> Fraction(Decimal('0.3')) % Fraction(6, 7)
    Fraction(3, 10)
    >>> Fraction(Decimal('0.3')) - Fraction(6, 7)
    Fraction(-39, 70)
    >>> Fraction(Decimal('0.3')) + Fraction(6, 7)
    Fraction(81, 70)
    >>> Fraction(Decimal('0.3')) * Fraction(6, 7)
    Fraction(9, 35)

    >>> from math import floor, ceil
    >>> ceil(Fraction('8/7'))
    2
    >>> floor(Fraction('8/7'))
    1
    ```

### number base conversions

bin() | oct() | hex()    | int(x, base)
------|-------|----------|------
base 2 | base 8 | base 16 | bases 2 to 36

* Example 

```python
>>> 0b0101
5
>>> 0o63527
26455
>>> 0o63
51
>>> 0xad2
2770
>>> hex(22)
'0x16'
>>> oct(22)
'0o26'
>>> bin(22)
'0b10110'
```

### complex

* complex construction string argument may have parentheses but must not contain spaces
* `cmath` standard Library module contains complex equivalents of math
* Example

    ```python
    >>> 1+1j
    (1+1j)
    >>> type(1+1j)
    <class 'complex'>
    >>> (1+1j) + (1-1j)
    (2+0j)
    >>> (1+1j) + (2-2j)
    (3-1j)
    >>> (1+1j) - (2-2j)
    (-1+3j)
    >>> complex('-2+1j')
    (-2+1j)
    >>> complex('(-2+1j)')
    (-2+1j)
    >>> complex(-2, 1)
    (-2+1j)


    ```


### date & time 
---

* Gregorian calendar
* `weekday()`


        0 Monday
        1 Tuesday
        2 Wednesday
        3 Thursday
        4 Friday
        5 Saturday
        6 Sunday
 

* `isoweekday()`

        1 Monday
        2 Tuesday
        3 Wednesday
        4 Thursday
        5 Friday
        6 Saturday
        7 Sunday   

* timedelta

    * Constructor accepts and sums
        • days
        • seconds
        • microseconds
        • milliseconds
        • minutes
        • hours
        • weeks
    * Instances store only
        • days
        • seconds
        • microseconds

* Example 

    ```python
    from datetime import (date, time)
    from datetime import datetime as Datetime
    from datetime import timedelta
    from datetime import (tzinfo, timezone)

    ### date

    >>> datetime.date(2000,month=2, day=10)
    datetime.date(2000, 2, 10)
    >>> datetime.date.today()
    datetime.date(2014, 3, 14)
    >>> datetime.date.fromtimestamp(99999999)
    datetime.date(1973, 3, 3)
    >>> datetime.date.fromordinal(9999)
    datetime.date(28, 5, 17)
    >>> datetime.date.max
    datetime.date(9999, 12, 31)
    >>> datetime.date.min
    datetime.date(1, 1, 1)
    >>> datetime.date.today().weekday()
    4
    >>> datetime.date.today().isoweekday()
    5
    >>> d = datetime.date.fromtimestamp(99999999)
    >>> d.strftime('%A %d %B %b')
    'Saturday 03 March Mar'
    >>> d.strftime('%A %d %B %b %Y')
    'Saturday 03 March Mar 1973'
    >>> "The date is {:%A %d %B %b %y}".format(d)
    'The date is Saturday 03 March Mar 73'
    >>> "{date:%A} {date.day} {date:%B} {date:%Y}".format(date=d)
    'Saturday 3 March 1973'

    ### time

    >>> t=datetime.time(23,59,1,7451)   
    >>> t.isoformat()
    '23:59:01.007451'
    >>> t.strftime('%Hh%Mm%Ss')
    '23h59m01s'
    >>> datetime.time.max
    datetime.time(23, 59, 59, 999999)
    >>> datetime.time.min
    datetime.time(0, 0)

    ### datetime

    >>> datetime.datetime(2001,6,7,8,15,25,895)
    datetime.datetime(2001, 6, 7, 8, 15, 25, 895)
    >>> dt= datetime.datetime(2001,6,7,8,15,25,895)
    >>> dt.isoformat()
    '2001-06-07T08:15:25.000895'

    ### timedelta
    >>> td = datetime.timedelta(weeks=2, days=1, hours=1, minutes=1, microseconds=2, milliseconds= 1)
    >>> td
    datetime.timedelta(15, 3660, 1002)

    ```<file_sep>+++
title = "F# Overview"
description = "F# Overview"
weight = 1 
+++

## F`#`

> F# is an open-source, cross-platform, interoperable programming language for writing succinct, robust and performant code. Your focus remains on your problem domain, rather than the details of programming.

### Organizing F# Code
The following table shows reference articles related to organizing your F# code.

Title	| Description
----------|------------
Namespaces |	Learn about namespace support in F#. A namespace lets you organize code into areas of related functionality by enabling you to attach a name to a grouping of program elements.
Modules	|	Learn about modules. An F# module is like a namespace and can also include values and functions. Grouping code in modules helps keep related code together and helps avoid name conflicts in your program.
open Declarations	|	Learn about how open works. An open declaration specifies a module, namespace, or type whose elements you can reference without using a fully qualified name.
Signatures	|	Learn about signatures and signature files. A signature file contains information about the public signatures of a set of F# program elements, such as types, namespaces, and modules. It can be used to specify the accessibility of these program elements.
Access Control	|	Learn about access control in F#. Access control means declaring what clients are able to use certain program elements, such as types, methods, functions, and so on.
XML Documentation	|	Learn about support for generating documentation files for XML doc comments, also known as triple slash comments. You can produce documentation from code comments in F# as in other .NET languages.

### Literals and Strings
The following table shows reference articles that describe literals and strings in F#.


Title	|	Description
----------|------------
Literals	|	Learn about the syntax for literal values in F# and how to specify type information for F# literals.
Strings	|	Learn about strings in F#. The string type represents immutable text, as a sequence of Unicode characters. string is an alias for System.String in .NET.
Interpolated strings	|	Learn about interpolated strings, a special form of string that allows you to embed F# expressions directly inside them.

### Values and Functions
The following table shows reference articles that describe language concepts related to values, let-bindings, and functions.

Title	|	Description
----------|------------
Values	|	Learn about values, which are immutable quantities that have a specific type; values can be integral or floating point numbers, characters or text, lists, sequences, arrays, tuples, discriminated unions, records, class types, or function values.
Functions	|	Functions are the fundamental unit of program execution in any programming language. An F# function has a name, can have parameters and take arguments, and has a body. F# also supports functional programming constructs such as treating functions as values, using unnamed functions in expressions, composition of functions to form new functions, curried functions, and the implicit definition of functions by way of the partial application of function arguments.
Function Expressions	|	Learn how to use the F# 'fun' keyword to define a lambda expression, which is an anonymous function.


### Loops and Conditionals
The following table lists articles that describe F# loops and conditionals.


Title	|	Description
----------|------------
Conditional Expressions: if...then...else	|	Learn about the if...then...else expression, which runs different branches of code and also evaluates to a different value depending on the Boolean expression given.
Loops: for...in Expression	|	Learn about the for...in expression, a looping construct that is used to iterate over the matches of a pattern in an enumerable collection such as a range expression, sequence, list, array, or other construct that supports enumeration.
Loops: for...to Expression	|	Learn about the for...to expression, which is used to iterate in a loop over a range of values of a loop variable.
Loops: while...do Expression	|	Learn about the while...do expression, which is used to perform iterative execution (looping) while a specified test condition is true.

### Pattern Matching
The following table shows reference articles that describe language concepts.


Title	|	Description
----------|------------
Pattern Matching	|	Learn about patterns, which are rules for transforming input data and are used throughout F#. You can compare data with a pattern, decompose data into constituent parts, or extract information from data in various ways.
Match Expressions	|	Learn about the match expression, which provides branching control that is based on the comparison of an expression with a set of patterns.
Active Patterns	|	Learn about active patterns. Active patterns enable you to define named partitions that subdivide input data. You can use active patterns to decompose data in a customized manner for each partition.

### Exception Handling
The following table shows reference articles that describe language concepts related to exception handling.

Title	|	Description
----------|------------
Exception Handling	|	Contains information about exception handling support in F#.
The try...with Expression	|	Learn about how to use the try...with expression for exception handling.
The try...finally Expression	|	Learn about how the F# try...finally expression enables you to execute clean-up code even if a block of code throws an exception.
The use Keyword	|	Learn about the keywords use and using, which can control the initialization and release of resources.
Assertions	|	Learn about the assert expression, which is a debugging feature that you can use to test an expression. Upon failure in Debug mode, an assertion generates a system error dialog box.

### Types and Type Inference
The following table shows reference articles that describe how types and type inference work in F#.


Title	|	Description
----------|------------
Types	|	Learn about the types that are used in F# and how F# types are named and described.
Basic Types	|	Learn about the fundamental types that are used in F#. It also provides the corresponding .NET types and the minimum and maximum values for each type.
Unit Type	|	Learn about the unit type, which is a type that indicates the absence of a specific value; the unit type has only a single value, which acts as a placeholder when no other value exists or is needed.
Type Abbreviations	|	Learn about type abbreviations, which are alternate names for types.
Type Inference	|	Learn about how the F# compiler infers the types of values, variables, parameters, and return values.
Casting and Conversions	|	Learn about support for type conversions in F#.
Generics	|	Learn about generic constructs in F#.
Automatic Generalization	|	Learn about how F# automatically generalizes the arguments and types of functions so that they work with multiple types when possible.
Constraints	|	Learn about constraints that apply to generic type parameters to specify the requirements for a type argument in a generic type or function.
Flexible Types	|	Learn about flexible types. A flexible type annotation is an indication that a parameter, variable, or value has a type that is compatible with type specified, where compatibility is determined by position in an object-oriented hierarchy of classes or interfaces.
Units of Measure	|	Learn about units of measure. Floating point values in F# can have associated units of measure, which are typically used to indicate length, volume, mass, and so on.
Byrefs	|	Learn about byref and byref-like types in F#, which are used for low-level programming.


### Tuples, Lists, Collections, Options
The following table shows reference articles that describe types supported by F#.

Title	|	Description
----------|------------
Tuples	|	Learn about tuples, which are groupings of unnamed but ordered values of possibly different types.
Collections	|	An overview of the F# functional collection types, including types for arrays, lists, sequences (seq), maps, and sets.
Lists	|	Learn about lists. A list in F# is an ordered, immutable series of elements all of the same type.
Options	|	Learn about the option type. An option in F# is used when a value may or may not exist. An option has an underlying type and may either hold a value of that type or it may not have a value.
Arrays	|	Learn about arrays. Arrays are fixed-size, zero-based, mutable sequences of consecutive data elements, all of the same type.
Sequences	|	Learn about sequences. A sequence is a logical series of elements all of one type. Individual sequence elements are only computed if necessary, so the representation may be smaller than a literal element count indicates.
Sequence Expressions	|	Learn about sequence expressions, which let you generate sequences of data on-demand.
Reference Cells	|	Learn about reference cells, which are storage locations that enable you to create mutable variables with reference semantics.

### Records and Discriminated Unions
The following table shows reference articles that describe record and discriminated union type definitions supported by F#.

Title	|	Description
----------|------------
Records	|	Learn about records. Records represent simple aggregates of named values, optionally with members.
Anonymous Records	|	Learn how to construct and use anonymous records, a language feature that helps with the manipulation of data.
Discriminated Unions	|	Learn about discriminated unions, which provide support for values that may be one of a variety of named cases, each with possibly different values and types.
Structs	|	Learn about structs, which are compact object types that can be more efficient than a class for types that have a small amount of data and simple behavior.
Enumerations	|	Enumerations are types that have a defined set of named values. You can use them in place of literals to make code more readable and maintainable.


### Object Programming
The following table shows reference articles that describe F# object programming.


Title	|	Description
----------|------------
Classes	|	Learn about classes, which are types that represent objects that can have properties, methods, and events.
Interfaces	|	Learn about interfaces, which specify sets of related members that other classes implement.
Abstract Classes	|	Learn about abstract classes, which are classes that leave some or all members unimplemented, so that implementations can be provided by derived classes.
Type Extensions	|	Learn about type extensions, which let you add new members to a previously defined object type.
Delegates	|	Learn about delegates, which represent a function call as an object.
Inheritance	|	Learn about inheritance, which is used to model the "is-a" relationship, or subtyping, in object-oriented programming.
Members	|	Learn about members of F# object types.
Parameters and Arguments	|	Learn about language support for defining parameters and passing arguments to functions, methods, and properties. It includes information about how to pass by reference.
Operator Overloading	|	Learn about how to overload arithmetic operators in a class or record type, and at the global level.
Object Expressions	|	Learn about object expressions, which are expressions that create new instances of a dynamically created, anonymous object type that is based on an existing base type, interface, or set of interfaces.

### Async, Tasks and Lazy
The following table lists topics that describe F# async, task and lazy expressions.

Title	|	Description
----------|------------
Async Expressions	|	Learn about async expressions, which let you write asynchronous code in a way that is very close to the way you would naturally write synchronous code.
Task Expressions	|	Learn about task expressions, which are an alternative way of writing asynchronous code used when interoperating with .NET code that consumes or produces .NET tasks.
Lazy Expressions	|	Learn about lazy expressions, which are computations that are not evaluated immediately, but are instead evaluated when the result is actually needed.

### Computation expressions and Queries
The following table lists topics that describe F# computation expressions and queries.

Title	|	Description
----------|------------
Computation Expressions	|	Learn about computation expressions in F#, which provide a convenient syntax for writing computations that can be sequenced and combined using control flow constructs and bindings. They can be used to manage data, control, and side effects in functional programs.
Query Expressions	|	Learn about query expressions, a language feature that implements LINQ for F# and enables you to write queries against a data source or enumerable collection.

### Attributes, Reflection, Quotations and Formatting
The following table lists articles that describe F# reflective features, including attributes, quotations, nameof, and plain text formatting.

Title	|	Description
----------|------------
Attributes	|	Learn how F# Attributes enable metadata to be applied to a programming construct.
nameof	|	Learn about the nameof operator, a metaprogramming feature that allows you to produce the name of any symbol in your source code.
Caller Information	|	Learn about how to use Caller Info Argument Attributes to obtain caller information from a method.
Source Line, File, and Path Identifiers	|	Learn about the identifiers __LINE__, __SOURCE_DIRECTORY__, and __SOURCE_FILE__, which are built-in values that enable you to access the source line number, directory, and file name in your code.
Code Quotations	|	Learn about code quotations, a language feature that enables you to generate and work with F# code expressions programmatically.
Plain Text Formatting	|	Learn how to use sprintf and other plain text formatting in F# applications and scripts.

### Type Providers
The following table lists articles that describe F# type providers.

Title	|	Description
----------|------------
Type Providers	|	Learn about type providers and find links to walkthroughs on using the built-in type providers to access databases and web services.
Create a Type Provider	|	Learn how to create your own F# type providers by examining several simple type providers that illustrate the basic concepts.

<file_sep>+++

title = "Closure & Decorator"
description ="Closure & Decorator"
weight=2
+++

## Closure & Decorator

### LEGB rules 
* Local, Enclosing, Gloable, Built-in


### Local function

* Useful for specialized, one-off functions
* Aid in code organization and readability
* Similar to lambdas, but more general
* May contain multiple expressions
* May contain statements

### Closure 
* Closure maintain references to objects from earlier scopes
* LEGB does not apply when making new bindings
* Usage of nonlocal

    * Example

        ```python
        def make_timer():
            last_called = None

            def elapsed():
                nonlocal last_called
                now = time.time()
                if last_called is None:
                    last_called = now
                    return None
                result = now - last_called
                last_called = now
                return result
            return elapsed


        if __name__ == "__main__":
            mt = make_timer()
            print(mt ())
            print('-----------------------------')
            print(mt ())
            print('-----------------------------')
            print(mt ())
        ```    

* Use as function factory

    * Example
    
        ```python

        def raise_to(exp):
            def raise_to_exp(x):
                return pow(x, exp)
            return raise_to_exp

        if __name__ == "__main__":
            square=raise_to(2)
            cube= raise_to(3)
            print(square(2))
            print(cube(2))

        ## test result:
        ## 4
        ## 8

        ```

### Decorator

* Replace, enhance, or modify existing functions
* Does not change the original function definition
* Calling code does not need to change
* Decorator mechanism uses the modified function’s original name
* Example

    * use function as decorator

        ```python
        def escape_unicode(f):
            def wrap(*args, **kwargs):
                x = f(*args, **kwargs)
                return ascii(x)
            return wrap


        @escape_unicode
        def hello_greek():
            return 'γειά σου κόσμος'    

        if __name__ == "__main__":
            print(hello_greek())

        ## test result:
        ## '\u03b3\u03b5\u03b9\u03ac \u03c3\u03bf\u03c5 \u03ba\u03cc\u03c3\u03bc\u03bf\u03c2'
        ```

    * Example: multiple decorators including function and instance

        ```python
        class Trace:
            def __init__(self):
                self.enabled = True

            def __call__(self, f):
                @functools.wraps(f)
                def wrap(*args, **kwargs):
                    if self.enabled:
                        print('Calling {}'.format(f.__name__))
                    return f(*args, **kwargs)
                return wrap

        def escape_unicode(f):
            @functools.wraps(f)
            def wrap(*args, **kwargs):
                x = f(*args, **kwargs)
                return ascii(x)
            return wrap

        @tracer
        @escape_unicode
        def hello_greek():
            return 'γειά σου κόσμος'    

        ## test result
        ## Calling hello_greek
        ## '\u03b3\u03b5\u03b9\u03ac \u03c3\u03bf\u03c5 \u03ba\u03cc\u03c3\u03bc\u03bf\u03c2'

        ```

    * Use as validator

        ```python
        def check_non_negative(index):
            def validator(f):
                def wrap(*args):
                    if args[index] < 0:
                        raise ValueError(
                            'Arg {} must be non-negative'.format(index)
                        )
                    return f(*args)
                return wrap
            return validator

        @check_non_negative(1)
        def create_seq(value, size):
            return [value]*size  

        if __name__ == "__main__":
            create_seq('0', -3)

        ## test result
        ....
        'Arg {} must be non-negative'.format(index)
        ValueError: Arg 1 must be non-negative
        ```

    ## Properties & Class

    ### @staticmethod

    * No access needed to either class or instance objects.
    * Most likely an implementation detail of the class.
    * May be able to be moved to become a module-scope function

    ### @classmethod

    * Requires access to the class object to call other class methods or the constructor
    * Always use self for the first argument to instance methods.
    * Always use cls for the first argument to class methods.
    * Use case: use as named constructors 

    ```python
    class FileStream(object):

        @classmethod
        def from_file(cls, filepath, ignore_comments=False, *args, **kwargs):   
            with open(filepath, 'r') as fileobj:
                for obj in cls(fileobj, ignore_comments, *args, **kwargs):
                    yield obj

        @classmethod
        def from_socket(cls, socket, ignore_comments=False, *args, **kwargs):
            raise NotImplemented ## Placeholder until implemented

        def __init__(self, iterable, ignore_comments=False, *args, **kwargs):
            ...
    ```

### @property

* Encapsulation

    * Example

        ```python
        class A
            @property
            def  prop(self):
                return self._prop

            @prop.setter
            def prop(self, value):
                self._prop = value
        ```
<file_sep>+++
title = "MySql: Schema & Metadata"
description="Query schema & permission "
+++

## Information schema

> INFORMATION_SCHEMA provides access to database metadata, information about the MySQL server such as the name of a database or table, the data type of a column, or access privileges. Other terms that are sometimes used for this information are data dictionary and system catalog. 

### Check out table size

```sql
SET @target_schema='THE_TARGET_SCHEMA';

SELECT 
  TABLE_NAME, table_rows, data_length, index_length,  
   round(((data_length + index_length) / 1024 / 1024 /1024),2) 'Size in GB',
  round(((data_length + index_length) / 1024 / 1024 ),2) 'Size in MB' 
FROM information_schema.TABLES
WHERE table_schema = @target_schema
ORDER BY data_length DESC
LIMIT 50;

```


### Check out running process 


```sql
SET @target_schema='THE_TARGET_SCHEMA';

SELECT * FROM information_schema.PROCESSES
WHERE command <> 'Sleep'
  AND db = target_schema 

;

-- Another short cut to show all process

SHOW FULL PROCESSLIST;

```


### Get the information of stored proc or function

```sql
SET @target_schema='THE_TARGET_SCHEMA';

SELECT * FROM information_schema.ROUTINES
WHERE  routine_schema = target_schema 
;

```


### Optimize table after deletion 


```sql
-- Query the table sorted by data free space
SELECT table_name , data_length, data_free
FROM information_schema.tables
WHERE table_schema=@target_schema 
AND data_free > 0
ORDER BY data_free DESC


-- Get table names which need optimization
SELECT table_name
FROM information_schema.tables
WHERE table_schema=@target_schema 
AND data_free > 0
ORDER BY data_free DESC


-- Optmize table 
OPTIMIZE TABLE XXXXX

```
<file_sep>+++
title = "Digital Ocean"
+++

{{%children style="card" description="true" sort="Weight" %}}<file_sep>+++
title = "Cron Job Note - 2"
description = "Common Cron Job examples - Refresh Cassandra database"
+++


### Refresh the database (NoSQL) - Cassandra

> The sample script is used to backup the data from production database and refresh the data to staging or test database. It is not supposed to restore data because of database corruption. 

#### Backup the Cassandra database nightly

* There is a keyspace named `hho_ks` in the Cassandra nodes store the production data. 
* Every night the staging Cassandra server will be refreshed with production's snapshot
* This solution is not built on the incremental snapshot.
* The production and staging nodes are running within the same subnet.
* The backup script will be run nightly on production server
* Cron job setting

    ```bash
    00 20   * * *   <user>  /home/<user>/bin/cass_snapshot.sh >> /home/<user>/bin/refresh.log 2>&1
    ```

* The script to create a snapshot:   `cass_snapshot.sh`


    ```bash
    #!/bin/bash

    # the log file sits home/<user>/bin/refresh.log 

    # SET staging Cassandra IP
    CASS_STG_IP=0.0.0.0

    echo "$(date): Beginning refresh of staging Cassandra ${CASS_STG_IP}"
    START=$(date +%s)

    cd /data/cassandra

    # Prepare a schema script of keyspace hho_ks
    echo "$(date): Prepare a schema script of keyspace hho_ks"
    sudo bash -c "cqlsh -e 'DESC KEYSPACE hho_ks' > hho_ks.cql"

    # Remove old snapshots folder and create new snapshots folder with some sub-folders
    echo "$(date): Remove old snapshots folder and create new snapshots folder with some sub-folders"
    sudo rm -rf snapshots
    sudo mkdir snapshots
    cd snapshots
    sudo mkdir -p service interval imported_file market_file meter_config
    cd /data/cassandra

    # Clear snapshot hho_ks
    echo "$(date): Clear snapshot hho_ks"
    # find /data/cassandra/data/hho_ks/  -name snapshots -type d
    nodetool clearsnapshot hho_ks > /dev/null 2>&1 
    # find /data/cassandra/data/hho_ks/  -name snapshots -type d

    # Create new snapshot (cut out the snapshot id into a variable)
    echo "$(date): Create new snapshot"
    SNAP_ID=$(nodetool snapshot hho_ks | cut -c 66-78)
    # echo SNAP_ID=$SNAP_ID

    # Copy snapshot data to new folder snapshot
    # ~ 7mins
    echo "$(date): Copy snapshot data to new snapshot folder"
    for d in $(ls /data/cassandra/snapshots); do \
    sudo cp -R /data/cassandra/data/hho_ks/$d*/snapshots/$SNAP_ID/. /data/cassandra/snapshots/$d/ ; \
    done

    # Create a tarball of snapshots
    echo "$(date): Create a tarball of snapshots"
    sudo rm -rf snapshots.tar.gz
    sudo tar -zcvf snapshots.tar.gz snapshots/ > /dev/null 2>&1  # ~30mins

    # Copy tarball and schema script to staging Cassandra
    echo "$(date): Copy tarball and schema script to staging Cassandra ${CASS_STG_IP}"
    scp snapshots.tar.gz <user>@${CASS_STG_IP}:/home/<user>/ # ~5mins
    scp hho_ks.cql <user>@${CASS_STG_IP}:/home/<user>/

    # Refresh snapshots on staging Cassandra
    echo "$(date): SSH to staging Cassandra ${CASS_STG_IP}"
    sudo ssh <user>@${CASS_STG_IP} 'bash -s' < /home/<user>/bin/cass_refresh.sh

    echo "$(date): Completed refresh of staging Cassandra ${CASS_STG_IP}"
    END=$(date +%s)
    echo "Refresh duration: $(( $END - $START ))s"

    ```

* Refresh the staging Cassandra node
* The script to refresh the staging node: `cass_refresh`


    ```bash
    #!/bin/bash

    # Unzip tarball
    echo "$(date): Unzip tarball"
    cd
    rm -rf snapshots
    tar -zxvf snapshots.tar.gz > /dev/null 2>&1 # ~5mins

    # Attempt to drop keyspace hho_ks 
    # It will likely throw a java.lang.RuntimeException
    echo "$(date): Drop keyspace hho_ks"
    cqlsh -e "drop keyspace hho_ks" > /dev/null 2>&1 
    sleep 1m

    # Start/restart Cassandra service
    echo "$(date): Restart Cassandra service"
    sudo systemctl start cassandra.service
    sudo systemctl restart cassandra.service

    # Wait for 5 minutes to ensure the Cassandra service is running
    echo "$(date): Wait for 5 minutes to ensure the Cassandra service is running"
    sleep 5m

    # Check Cassandra service status
    # sudo systemctl status cassandra

    # Attempt to drop keyspace hho_ks again 
    # It will likely complain: Cannot drop non existing keyspace 'hho_ks'
    cqlsh -e "drop keyspace hho_ks" > /dev/null 2>&1 
    sleep 1m

    # Remove data folder hho_ks
    echo "$(date): Remove data folder hho_ks"
    sudo rm -r -f /var/lib/cassandra/data/hho_ks

    # Recreate keyspace hho_ks
    echo "$(date): Recreate keyspace hho_ks"
    cqlsh --file="hho_ks.cql" > /dev/null 2>&1 

    # Copy snapshots to new data folder hho_ks
    # ~5mins
    echo "$(date): Copy snapshots to new data folder hho_ks"
    for d in $(ls snapshots); do \
    sudo cp -R snapshots/$d/. /var/lib/cassandra/data/hho_ks/$d*/ ; \
    done; 

    # Refresh keyspace
    # ~10mins
    echo "$(date): Refresh keyspace"
    for d in $(ls snapshots); do \
    nodetool refresh -- hho_ks $d ; \
    done


    ```

<file_sep>+++
title = "DigitialOcean: Get Started"
description="User Setup, Security Update & Features"
weight=2
+++

## User Setup

### Create a new admin user

#### Add a new user 

```
# Add new user 
# set password
adduser <admin_user>

# user to sudo group
usermod -aG sudo <admin_user>

```

#### Set SSH access for new user

```
# Switch session to new user 
su - <admin_user>

# navigate to user home 
cd

# Prepare ssh directory
mkdir .ssh
chmod 700 ~/.ssh

# Copy root key
sudo cp /root/.ssh/authorized_keys ~/.ssh/authorized_keys
chmod 644 /home/<admin_user>/.ssh/authorized_keys
sudo chown -R <admin_user>:<admin_user> ~/
```

#### Login as new user via SSH 

```
ssh -i ~/.ssh/<your_droplet_rsa> <admin_user>@<your_droplet_ip>
```

#### Set root password & disable SSH

```
sudo passwd

# rename key file
sudo mv /root/.ssh/authorized_keys  /root/.ssh/disabled_authorized_keys
```


### Security Update


```
# Update only for security
sudo apt-get install unattended-upgrades

# Update security packages
sudo unattended-upgrade -d --dry-run
sudo unattended-upgrade -d

# Update quietly
sudo unattended-upgrade
```


### Tagging & Cloud Firewall

* Tags are custom labels you apply to Droplets that have multiple uses
  * Add tags to your droplet. e.g. my-web-server
* DigitalOcean Cloud Firewalls are a free, network-based, stateful firewall service for your DigitalOcean Droplets. They block all traffic that isn’t expressly permitted by a rule. You can define the Droplets protected by a firewall individually or by using tags.
* Always setup Firewall for your droplets
  * Set SSH permission for only given IP address
  * Set HTTP for port 80
  * Set HTTPS for port 443




## Other Features 

### Floating IPs

DigitalOcean Floating IPs are publicly-accessible static IP addresses that you can assign to Droplets. A floating IP provides an additional static address you can use to access a Droplet without replacing or changing the Droplet’s original public IP address.

### Block Storage Volumes

DigitalOcean Block Storage is a flexible, convenient way of managing additional storage (in units called volumes) for your Droplets. Volumes are independent resources that you can move between Droplets within the same region. You can increase the size of a volume without powering down the Droplet it’s attached to. They’re most useful when you need more storage space but don’t need the additional processing power or memory that a larger Droplet would provide,


### Load Balancers
DigitalOcean Load Balancers are a fully-managed, highly available load balancing service. Load balancers distribute traffic to groups of Droplets, which decouples the overall health of a backend service from the health of a single server to ensure that your services stay online.

## Known Limits

* Some Droplet network traffic is restricted to help prevent malicious actions, like reflected DDoS attacks

  * TCP and UDP traffic on port 11211 inbound from external networks (due to the Memcached amplification attacks in March 2018)
  * Multicast traffic.
  * Traffic not matching a Droplet's IP address/MAC address.
  * SMTP via Floating IPs and IPv6.

* Users can create up to 100 volumes and up to a total of 16 TiB of disk space per region. You can contact our support team to request an increase. You can attach a maximum of 7 volumes to any one node or Droplet, and this limit cannot be changed.

* General Purpose plans are not yet compatible with DigitalOcean Kubernetes or Managed Databases.

* You can't create more than 10 Droplets at the same time using the control panel or the API


### Build Web Host

* Next step is to build a web host in droplet. <file_sep>+++
title = "Python"
description = "Python Notes"
+++

{{%children style="card" description="true" sort="Weight" %}}<file_sep>+++
title = "AWS"
+++

{{%children style="card" description="true" sort="Weight" %}}<file_sep>+++
title = "Java"
description = "Java Notes"
+++

{{%children style="card" description="true" sort="Weight" %}}<file_sep>+++

title = "Debug PHP with Free IDE"
description="Eclipse PDT and Netbeans for PHP development"
+++

## PDT and Netbeans

> *PDT and Netbeans are two most popluar free PHP IDEs. We choose such IDE for productivity, so code intelligence and debug are two key factors, which let us love IDE. Because both are not created for PHP development at the start, there is no built-in server to support the PHP web debug. When we want to use it to debug, we would come across some wierd problems. Here is how to prepare the IDE for PHP debug.*

## AMP package or without AMP

### Use AMP package as smart choice 

* If you haven't installed install any MySQL, Apache, PHP on your computer, I will recommend you to choose AMP ( Apache, MySQL, PHP ) package first, especially when project is small, the time is so tight, your client just want you to do some minor change. In that scenario, AMP package is a much better choice. Popular AMP packages include : EasyPHP, MAMP, WAMP, XAMPP. You can pick any of them on your favor.  

* Here is not going to discuss any specific AMP package. You can find more detailed instruction from their official website. If you still have problem, you can take a look how to do without the AMP package, but I won't guarantee you the solution below will work for your AMP.  

### Manage your shits without AMP package

* Here we just focus on how to work with shits ( Apache, PHP ) which you download and install them piece by piece. 10 years ago, it was easier to choose which one to download and install, because there was no much option, but now there are many options which make us confused.  

* The more worst and nastiest problem today is compatible issue between x86 and x64 applications in Windows. Even sometimes they are claimed compiled as x86, until you test it you will never they are really compatible. The reason is Windows has different versions of C++ redistributed compiler, if you use different compilers from Windows to compile your source code, you cannot ensure they are compatible to work together. 

* To avoid this problem, it is better to make sure the package or software are compatible at the beginning. That is why the AMP package is much better and easy to do that. They help you solve such nasty problem by bundling all you need together. 

* Download the compatible packages, especially Apache and PHP. When you download PHP, you need to know the PHP is compiled by VC9, VC11, or VC14, and x86 or x64. After that, you need to download proper Apache from [here](https://www.apachelounge.com/download)

## Prerequisites

* Apache path `c:\apache`. Version 2.4.x, VC11, x86
* PHP path `c:\php`, Version 5.6.x, VC11, x86
* Use `localhost:1234` as test website URL
* Project workspace path `c:\php_workspace`
* Website root path `c:\php_workspace\phpsite`, the `index.php` is under this root path


## Prepare PHP for debugging

### Download XDebug and install it

* Go to xdebug [site](xdebug.org). Use the [wizard tool](https://xdebug.org/wizard.php) to find the xdebug tool. 

* Type `c:\php\php.exe -i | clip` to copy the php info to memory. And then paste the content to the input area, and click `Analyse my phpinfo() output`. It will show the correct file to download. 

* Download the dll file and put it into php folder `c:\php\ext`. Update the `php.ini` file by adding the following lines at the bottom of file. 

```ini
[XDebug]
zend_extension="c:/php/ext/php_xdebug-x.x.x-x.x-vcxx.dll"
xdebug.remote_enable=1
xdebug.remote_host=localhost
xdebug.remote_port=9000
xdebug.remote_autostart=1
xdebug.remote_connect_back=1
```

### Setup Apache to load PHP

* Add php module loading inside your apache configuration file. 
* On the file `c:\apache\conf\httpd.conf` with nodepad and update as following setting

```apache

### Update apache root
### ServerRoot "c:/Apache24"
ServerRoot "c:/apache"

### Change origin 80 to 1234
### Listen 80
Listen 1234

### Add ServerName
ServerName localhost:1234

### Add PHP directory
PHPIniDir "C:/php"

### Add PHP module and handler
LoadModule php5_module "c:/php/php5apache2_4.dll"
AddHandler application/x-httpd-php .php

<FilesMatch \.php$>
      SetHandler application/x-httpd-php
</FilesMatch>

### Change origin doc root htdocs 
### DocumentRoot "c:/Apache24/htdocs"
DocumentRoot "c:/php_workspace/phpsite"

<Directory "c:/php_workspace/phpsite">
    Options Indexes FollowSymLinks
    AllowOverride All
    Require all granted
</Directory>
```


## Debug PHP with PDT 

**If you have PHP 7 installed, please choose the up to PHP 5.6.x as PHP runtime.**

* Open phpsite as PHP the project with Eclipse PDT 

* Setup PHP Web Application for debugging

  - Choose menu `Run` > `Debug Configurations` > `PHP Web Application` 
  - Add new configuration by clicking ![New](/img/php_pdt_new_debug.png)

* Configure PHP Web Server

  - Choose `Default PHP Web Server` from the dropdown list
  - Click the button  `configuration`, it prop up a Window dialog. 
  - On the tab `Server`, Set the `localhost:1234` as `Base URL`. It should be the same as ServerName in your `httd.conf` 
  - On the tab `Debugger`, choose `XDebug` from the dropdown list, then other setting as default. 
  - On the tab `Path Mapping`, add new mapping. Enter `/` as `Path on Server`, Put `c:\php_workspace\phpsite` as `Path in File system`, then leave other setting as default.
  - Close the Window dialog.
  - Choose the File `c:\php_workspace\phpsite\index.php` as startup page.
  - If the `Auto Generated URL` is not `localhost:1234/index.php`, then manually update it.
  - After all these done, you can debug your website now. 


## Debug PHP with Netbeans 

**If you have PHP 7 installed, please choose the up to PHP 5.6.x as PHP runtime.**

* Open phpsite as PHP the project with Netbeans. 

* Configure PHP Web Server

  - On the `Projects` panel, choose the project `phpsite` , right click and choose `Properties` 
  - Choose `Sources` within the categories. Check the PHP version is the same as your PHP version. 
  - Choose `Run Configurations` within the categories, and update the default configuration.
  - Choose `Local Web Site` from `Run As` dropdown list.
  - Set `localhost:1234` as Project URL
  - Click the button `Advanced ...` to update web server
  - Add a new path mapping. Enter `/` as `Path on Server`, Put `c:\php_workspace\phpsite` as `Path in File system`, then leave other setting as default.
  - Leave other default setting and click button `OK`
  - Now you can debug php site with Netbeans


## Use Nginx instead of Apache

* Download `RunHiddenConsole` 
  - Download [RunHiddenConsole](http://redmine.lighttpd.net/attachments/660/RunHiddenConsole.zip)
  - Extract the file `RunHiddenConsole.exe` to folder `c:\bin\`

* Install Nginx 32 bit version. 
  - We assume the ngnix's path is `c:\nginx\`

* Confirm `php-cgi.exe` is within the PHP folder `c:\php`.

* Setup Nginx FastCGI with PHP

  - Back the original `nginx.conf` 
  - Create a script to launch `nginx` and `php` in sequence.

```dos
@ECHO OFF
ECHO Start PHP FastCGI...
SET PATH=c:\php;%PATH%
c:\bin\RunHiddenConsole.exe c:\php\php-cgi.exe -b 127.0.0.1:9000
ECHO Start Nginx ...
c:\bin\RunHiddenConsole.exe c:\nginx\nginx.exe 
```

  - Open the `nginx.conf` via notepad
  - Replace the `server` block with following setting

```nginx
server {
        listen       1234;
        server_name  localhost;
        root          c:/php_workspace/phpsite;
        #charset koi8-r;

        ### Static
        location / {
            index  index.php;
        ###    try_files $uri $uri/ @missing;
        }
        location ~ /\.ht {
            deny  all;
        }
        location ~ /\.rewrite {
            deny  all;
        }
        
        ### PHP FastCGI
        location ~ \.php$ {
            root           c:/php_workspace/phpsite;
            ### root  html; 
            fastcgi_pass   127.0.0.1:41234;
           fastcgi_index  index.php;
           fastcgi_param  SCRIPT_FILENAME  c:/php_workspace/phpsite/$fastcgi_script_name;
           include        fastcgi_params;
        }

}
``` 


cat file | xclip -selection clipboard


### PHP 5.x anp PHP 7.x on Ubuntu 16

## Add repo
    sudo add-apt-repository -y ppa:ondrej/php
    sudo apt-get update
    sudo apt-get install php5.6-fpm
    sudo apt-get install 


### Trouble shooting


## Update
Today I got again problem with PHP 7 running despite I have disabled php7.0 apache module: phpinfo was showing php 7 using fastCGI ...
... So if after you follow the below instructions you face this situation, you may need to disable the proxy_fcgi apache module:

    sudo a2dismod proxy_fcgi proxy; sudo service apache2 restart

1. Re-Install PHP 5.6

What worked for me was this guide: http://www.lornajane.net/posts/2016/php-7-0-and-5-6-on-ubuntu

Actually is not required to remove php7.0, you can install php5.6 together ( also because you will have dependency problem with phpmyadmin package that required php7.0)

Assuming libapache2-mod-php is a suitable way to enable PHP in Apache for you, you can proceed in this way:

    sudo add-apt-repository ppa:ondrej/php
    sudo apt-get update
    sudo apt-get install php7.0 php5.6 php5.6-mysql php-gettext php5.6-mbstring \
        php-mbstring php7.0-mbstring php-xdebug libapache2-mod-php5.6 libapache2-mod-php7.0

2. Switch PHP version:

From php5.6 to php7.0:

Apache:

    sudo a2dismod php5.6 ; sudo a2enmod php7.0 ; sudo service apache2 restart

CLI:

    sudo update-alternatives --set php /usr/bin/php7.0

From php7.0 to php5.6:

Apache:

    sudo a2dismod php7.0 ; sudo a2enmod php5.6 ; sudo service apache2 restart

CLI:

    sudo update-alternatives --set php /usr/bin/php5.6



## Build xdebug for different PHP

### PHP 5.6 

```
php5.6 -i | xsel --clipboard

### open url http://xdebug.org/wizard.php
### copy the content and download the correct xdebug tar ball xdebug-2.5.3.tar.gz

tar -xvf xdebug-2.5.3.tar.gz 

cd xdebug-2.5.3

phpize5.6 

### You will output as below
###  ...
### Zend Module Api No:      20131226
### Zend Extension Api No:   220131226

./configure --with-php-config=/usr/bin/php-config5.6

make

sudo cp modules/xdebug.so /usr/lib/php/20131226


```

### Create xdebug.ini with `mods-available`

```
zend_extension="/usr/lib/php/20131226/xdebug.so"
xdebug.remote_enable=1
xdebug.remote_handler=dbgp 
xdebug.remote_mode=req
xdebug.remote_host=127.0.0.1 
xdebug.remote_port=9000
```

### Create symbolic links

```
sudo ln -s /etc/php/5.6/mods-available/xdebug.ini /etc/php/5.6/cli/conf.d/20-xdebug.ini
sudo ln -s /etc/php/5.6/mods-available/xdebug.ini /etc/php/5.6/fpm/conf.d/20-xdebug.ini
```





### PHP 7.0

```

php7.0 -i | xsel --clipboard

### open url http://xdebug.org/wizard.php
### copy the content and download the correct xdebug tar ball xdebug-2.5.3.tar.gz

tar -xvf xdebug-2.5.3.tar.gz 

cd xdebug-2.5.3

phpize7.0 

### You will output as below
### ...
### Zend Module Api No:      20151012
### Zend Extension Api No:   320151012

./configure --with-php-config=/usr/bin/php-config7.0

make

sudo cp modules/xdebug.so /usr/lib/php/20151012

```


### Create xdebug.ini with `mods-available`

```
zend_extension="/usr/lib/php/20151012/xdebug.so"
xdebug.remote_enable=1
xdebug.remote_handler=dbgp 
xdebug.remote_mode=req
xdebug.remote_host=127.0.0.1 
xdebug.remote_port=9000
```

### Create symbolic links

```
sudo ln -s /etc/php/7.0/mods-available/xdebug.ini /etc/php/7.0/cli/conf.d/20-xdebug.ini
sudo ln -s /etc/php/7.0/mods-available/xdebug.ini /etc/php/7.0/fpm/conf.d/20-xdebug.ini
```





<file_sep>+++
title = "AWS: SQS,SNS,SES - 2"
description = "Use Case - SQS, SNS, SES"
weight=7
+++

## Use Case

### Overview

{{<mermaid>}}
graph LR
    Sender_Email("<EMAIL>")
    Email_Failed
    Email_Delivered
    SNS_Subscriptions --> Email_Failed
    SNS_Subscriptions --> Email_Delivered
    Bounce_Notification --> Email_Failed
    Complaint_Notification --> Email_Failed
    Delivery_Notification --> Email_Delivered
    subgraph SQS     
        subgraph Email_Status_Queue
            SNS_Subscriptions
        end
    end 
    subgraph SNS
       subgraph Topics
           Email_Failed
           Email_Delivered
       end
    end
    subgraph SES
       Sender_Email
       subgraph Notifications
           Bounce_Notification 
           Complaint_Notification 
           Delivery_Notification 
       end
    end

{{</mermaid >}}


### SNS Setup

* Create a topic for failed email, e.g. bounce or spam complaint
    - It is named **Email_Failed** in the diagram above

* Create a topic for delivered email
    - It is named **Email_Delivered** in the diagram above


### SES Setup

* Create a new domain for sender email, e.g. <EMAIL>
  
* Setup the notifications
    - Bounce Notification maps to **Email_Failed**
    - Complaint Notification maps to **Email_Failed**
    - Delivery Notification maps to **Email_Delivered**
  
* Verify the domain - test.com

* Verify the DKIM - *.domainkey.test.com

### SQS Setup

* Create a new queue named **Email_Status_Queue**
* Add SNS subscriptions **Email_Failed** and **Email_Delivered** to the queue
  

### Integration

* Sender code of sample

```java
package email.sample;

// import ....

public class SesSample {

    static final String FROM = "<EMAIL>";

    static final String TO = "<EMAIL>";

    static final String CONFIGSET = "ConfigSet";

    // The subject line for the email.
    static final String SUBJECT = "SES test";

    // The email body for recipients with non-HTML email clients.
    static final String TEXTBODY = "This email was sent through Amazon SES "

    public static void main(String[] args) throws IOException {

        try {
            AmazonSimpleEmailService client = 
                AmazonSimpleEmailServiceClientBuilder.standard()
                // Replace the AWS Region
                    .withRegion(Regions.US_WEST_2).build();
            SendEmailRequest request = new SendEmailRequest()
                .withDestination(
                    new Destination().withToAddresses(TO))
                .withMessage(new Message()
                    .withBody(new Body()
                        .withText(new Content()
                            .withCharset("UTF-8").withData(TEXTBODY)))
                    .withSubject(new Content()
                        .withCharset("UTF-8").withData(SUBJECT)))
                .withSource(FROM);
            client.sendEmail(request);
            System.out.println("Email sent!");
            } catch (Exception ex) {
            System.out.println("Error message: " + ex.getMessage());
        }
    }
}

```

* Sample of SQS code

```java
// .....
public class SqsConsumer {
    
      public void receive(Object message) throws Exception {

        if (message instanceof CamelMessage) {
            String body = ((CamelMessage) message).getBodyAs(String.class, camelContext());
            JsonNode envelope = Json.parse(body);
            if (envelope.has("Message")) {
                JsonNode notification = Json.parse(envelope.get("Message").asText());
                String notificationType = notification.get("notificationType").asText();

                log.debug("Processing email notification: " + notificationType);
                switch (notification.get("notificationType").asText()) {
                    case "Received":
                        received.tell(new EmailActorProtocol.EmailReceived(notification), self());
                        break;
                    case "Bounce":
                        response.tell(new EmailActorProtocol.EmailBounced(notification), self());
                        break;
                    case "Delivery":
                        response.tell(new EmailActorProtocol.EmailDelivered(notification), self());
                        break;
                    case "Complaint":
                        response.tell(new EmailActorProtocol.EmailComplaintReceived(notification), self());
                        break;
                    default:
                        throw new RuntimeException(String.format("Notification type %s not supported", notificationType));
                }
            }
        }
    }
}

// EmailActorProtocol 
// public class EmailActorProtocol {
//     public interface EmailResponse {
//     }
//     @Data
//     static public class EmailReceived {
//         private final JsonNode body;
//     }
//     @Data
//     static public class EmailDelivered implements EmailResponse {
//         private final JsonNode body;
//     }
//     @Data
//     static public class EmailBounced implements EmailResponse {
//         private final JsonNode body;
//     }
//     @Data
//     static public class EmailComplaintReceived implements EmailResponse {
//         private final JsonNode body;
//     }
//     static public class GetHealth {
//     }
//     @Data
//     static public class Health {
//         private final boolean healthy;
//     }
// }


```


<file_sep># Blog

[![Join the chat at https://gitter.im/blog-hugo/Lobby](https://badges.gitter.im/blog-hugo/Lobby.svg)](https://gitter.im/blog-hugo/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)


* Use any `markdown` editor to edit the blog or post under the folder `content`
* Commit the markdown files to repo `blog-hugo` by running script the commit.bat. The commit.bat will only commit the markdown files. To commit other files please use git bash or git extension.
* Deploy posts to harryho.github.io by running script deploy.bat. After that, you can check it via browser.

#### __Hugo version upto 0.54__

> The hugo binaries can be found under folder `archived/hugo`

The theme Docdock has been forked as customized repository 

```
git submodule add --force https://github.com/harryho/hugo-theme-docdock.git themes/docdock 

```

Update submodule 

```
git submodule init
git submodule update

cd themes/<your_theme>
git checkout hho-blog  # only hho-blog branch should be used
git pull

```


Run the site locally

```
hugo server

# with script 
# Linux / Mac
./start.sh

# Windows
./start.cmd
```


Edit the site with draft

```bash
# Run the start script with param d
./start.sh  d
```
<file_sep>+++
date = "2018-07-04T14:59:31+11:00"
title = "Ubuntu Desktop 18 LTS note"
description = "Post-installation for Ubuntu 18 desktop"
+++

### Prelude

> This article is mainly to record the stuff to do post Ubuntu Desktop 18.04 Installation.


### Purpose 

All actions post installation is to make the Ubuntu Desktop a wonderful toolkit for developer. 

### Prerequisite

* Install all essentials

```bash
sudo apt install -y git curl
sudo apt-get install -y apt-transport-https ca-certificates      gnupg-agent
sudo apt install -y software-properties-common 
```

### Install & Setup Zsh

* Install Zsh

```bash
sudo apt install -y zsh
```

* Setup Zsh

> I prefer [Prezto Zsh](https://github.com/sorin-ionescu/prezto.git), which is the minimal version of Oh-My-Zsh. In my opinion, Oh-My-Zsh is kind of slow and sort of overblown.


- Create a script `setup_prezto.zsh`
- Save the code below to the scirpt and run `zsh ./setup_prezto.sh && source ~/.zshrc `

```bash

git clone --recursive https://github.com/sorin-ionescu/prezto.git \
    "${ZDOTDIR:-$HOME}/.zprezto"

setopt EXTENDED_GLOB
for rcfile in "${ZDOTDIR:-$HOME}"/.zprezto/runcoms/^README.md(.N); do
    ln -s "$rcfile" "${ZDOTDIR:-$HOME}/.${rcfile:t}"
done


if [ -d ~/.zprezto ];then
    cp  ubt18/z* ~/.zprezto/runcoms/
fi

[[ ! -d ~/.zprezto-contrib ]] && mkdir -p ~/.zprezto-contrib;

sudo -R $UsER:$USER .zprezto
sudo -R $UsER:$USER .zprezto-contrib

chsh -s /bin/zsh

echo "Setup finished."
echo "Please reboot or restart terminal."

```

### Install & Setup Vim with useful plugins

* Install Vim

```
sudo apt install -y vim
```

* Setup Vim with some amazing plugins

> There are many different vim plugins available online. There is a plance callded [Vim Awesome](https://vimawesome.com/), which you can find anything you want. 

> My favorite option is the [Junegunn's Vim plugins](https://github.com/junegunn/vim-plug). I like the simplicity of this solution.

- Prepare the vimrc as below 

```
set nu

colorscheme delek

:imap jj <Esc>
:imap jk <Esc>
:imap kj <Esc>
:imap ii <Esc>

" Specify a directory for plugins
" - For Neovim: ~/.local/share/nvim/plugged
" - Avoid using standard Vim directory names like 'plugin'
call plug#begin('~/.vim/plugged')

" Make sure you use single quotes

" Shorthand notation; fetches https://github.com/junegunn/vim-easy-align
Plug 'junegunn/vim-easy-align'

" Any valid git URL is allowed
Plug 'https://github.com/junegunn/vim-github-dashboard.git'

" Multiple Plug commands can be written in a single line using | separators
Plug 'SirVer/ultisnips' | Plug 'honza/vim-snippets'

" On-demand loading
Plug 'scrooloose/nerdtree', { 'on':  'NERDTreeToggle' }
Plug 'tpope/vim-fireplace', { 'for': 'clojure' }

" Using a non-master branch
Plug 'rdnetto/YCM-Generator', { 'branch': 'stable' }

" Using a tagged release; wildcard allowed (requires git 1.9.2 or above)
Plug 'fatih/vim-go', { 'tag': '*' }

" Plugin options
Plug 'nsf/gocode', { 'tag': 'v.20150303', 'rtp': 'vim' }

" Plugin outside ~/.vim/plugged with post-update hook
Plug 'junegunn/fzf', { 'dir': '~/.fzf', 'do': './install --all' }
Plug 'Shougo/vimproc.vim', { 'do': 'make' }


function! BuildYCM(info)
  " info is a dictionary with 3 fields
  " - name:   name of the plugin
  " - status: 'installed', 'updated', or 'unchanged'
  " - force:  set on PlugInstall! or PlugUpdate!
  if a:info.status == 'installed' || a:info.force
    !./install.py
  endif
endfunction

Plug 'Valloric/YouCompleteMe', { 'do': function('BuildYCM') }

" Plug 'Valloric/YouCompleteMe', { 'do': './install.py' }

Plug 'fatih/vim-go', { 'do': ':GoInstallBinaries' }

" Unmanaged plugin (manually installed and updated)
Plug '~/my-prototype-plugin'

" Initialize plugin system
call plug#end()
```

### Install MySql

- Here I just install the default MySql 5.7. If you want to install the new version or MariaDB, please check out the official website. 


```bash
# Install MySql server and client
sudo apt install mysql-server mysql-client

# Check if the MySql service active and running
sudo systemctl status mysql.service

# Enable or Restart MySql service
sudo systemctl enable mysql.service
sudo systemctl restart mysql.service

# Create root account
sudo mysql_secure_installation

# After the password for root has been created
sudo mysql -u root -p

# Update the password via mysql comomand pompt
mysql>ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY '<PASSWORD>';

# Create another admin is highly recommended in production env
mysql>GRANT ALL PRIVILEGES ON *.* TO 'admin'@'localhost' IDENTIFIED BY 'your_strong_password';
```



### Install Java

* Install [AdoptOpenJDK](https://adoptopenjdk.net/installation.html#x64_linux-jdk) for Linux

    ```
    # Download tarball you need
    # Unzip the tarball

    tar xzf OpenJDK11U-jdk_x64_linux_hotspot_x.y.z.tar.gz

    sudo mv OpenJDK11U-jdk_x64_linux_hotspot_x.y.z.tar.gz  \
        /usr/lib/jvm/OpenJDK11U-jdk_x64_linux_hotspot_x.y.z

    sudo apt install update-java-alternatives

    # Check alternative JDK 
    install update-java-alternatives -l

    adoptopenjdk-11-hotspot-amd64  1111   /usr/lib/jvm/adoptopenjdk-11-hotspot-amd64
    adoptopenjdk-14-hotspot-amd64  1141   /usr/lib/jvm/adoptopenjdk-14-hotspot-amd64
    adoptopenjdk-8-hotspot-amd64   1081   /usr/lib/jvm/adoptopenjdk-8-hotspot-amd64

    # Switch JDK
    update-java-alternatives -s adoptopenjdk-xx-hotspot-amd64

    ```



### Other useful tools

- Office
    - I prefer [WPS Office](http://wps-community.org). Rename the template files, because the template names contain Chinese characters and it may cause problem later.
    

- Dictionary
    - Offline Dictionary
    
    
    ```
    ## Install dict client & server (dictd)
    sudo apt install -y dict
    sudo apt install -y dictd

    ## Install dictionary libraries
    sudo apt-get install dict-gcide
    sudo apt-get install dict-wn
    sudo apt-get install dict-devil
    sudo apt-get install dict-moby-thesaurus
    ```

- Meida

    - VLC
    
    ```
    sudo apt install vlc
    ```

    - Spotify

    ```bash
    sudo snap install spotify
    ```














<file_sep>+++

title = "Java Note - 2: Concurrency"
description="Common good practice for Java Concurrency programming"
+++

## Thread

### Join
* The join method allows one thread to wait for the completion of another. join responds to an interrupt by exiting with an InterruptedException.

* Demo code of thread join

```java
public class JoinDemo implements Runnable {
    private Random rand = new Random(System.currentTimeMillis());

    public void run() {
        //simulate some CPU expensive task
        for (int i = 0; i < 100000000; i++) {
            rand.nextInt();
        }
        System.out.println("[" + Thread.currentThread().getName() + "] finished  .");
    }

    public static void main(String[] args) throws InterruptedException {
        Thread[] threads = new Thread[5];
        for (int i = 0; i < threads.length; i++) {
            threads[i] = new Thread(new JoinDemo(), "joinThread " + i);
            threads[i].start();
        }
        for (int i = 0; i < threads.length; i++) {
            threads[i].join();
        }
        System.out.println("[" 
            + Thread.currentThread().getName() 
            + "] All  -threads have finished.");
    }
}
```

### Common problem of multithred program 

* When there are many threads running, the exact sequence in which all running threads are executed depends next to the thread
configuration like priority also on the available CPU resources and the way the scheduler chooses the next thread to execute.
Although the behavior of the scheduler is completely deterministic, it is hard to predict which threads execute in which moment
at a given point in time. This makes access to shared resources critical as it is hard to predict which thread will be the first thread
that tries to access it.

* Sample code without sync can show you what the problem is. If you run the following sample code, you may get different output from mine here. It is also common Thread safe issue for multiple threads program. 

```java
public class NotSyncCounter implements Runnable {
    private static int counter = 0;

    public void run() {
        while (counter < 10) {
            System.out.println("[" 
                + Thread.currentThread().getName() 
                + "]  -    before: " + counter);
            counter++;
            System.out.println("[" 
                + Thread.currentThread().getName() 
                + "]  -    after: " + counter);
        }
    }

    public static void main(String[] args) throws InterruptedException {
        Thread[] threads = new Thread[5];
        for (int i = 0; i < threads.length; i++) {
            threads[i] = new Thread(new NotSyncCounter(), "  -    thread-" + i);
            threads[i].start();
        }
        for (int i = 0; i < threads.length; i++) {
            threads[i].join();
        }
    }
}


//Possible output:
// [  -    thread-2]  -    before: 0
// [  -    thread-1]  -    before: 0
// [  -    thread-4]  -    before: 0
// [  -    thread-3]  -    before: 0
// [  -    thread-0]  -    before: 0
// [  -    thread-3]  -    after: 4
// [  -    thread-3]  -    before: 5
// [  -    thread-4]  -    after: 3
// [  -    thread-1]  -    after: 2
// [  -    thread-1]  -    before: 6
// [  -    thread-1]  -    after: 7
// [  -    thread-2]  -    after: 1
// [  -    thread-1]  -    before: 7
// [  -    thread-4]  -    before: 6
// [  -    thread-4]  -    after: 9
// [  -    thread-4]  -    before: 9
// [  -    thread-3]  -    after: 6
// [  -    thread-0]  -    after: 5
// [  -    thread-4]  -    after: 10
// [  -    thread-1]  -    after: 8
// [  -    thread-2]  -    before: 7
// [  -    thread-2]  -    after: 11
```

* To solve the problme, there is `synchronized` keyword in Java available for us to handle the multiple threads program. 

* Demo code of `synchronized` to solve the problem on above sample.

```java

public class SyncCounter implements Runnable {
    private static int counter = 0;

    public void run() {
        while (counter < 10) {
            synchronized (SyncCounter.class) {
                System.out.println("[" 
                    + Thread.currentThread().getName() 
                    + "] - before: " + counter);
                counter++;
                System.out.println("[" 
                    + Thread.currentThread().getName() 
                    + "] - after: " + counter);
            }
        }
    }

    public static void main(String[] args) throws InterruptedException {
        Thread[] threads = new Thread[5];
        for (int i = 0; i < threads.length; i++) {
            threads[i] = new Thread(new SyncCounter(), "  -    thread-" + i);
            threads[i].start();
        }
        for (int i = 0; i < threads.length; i++) {
            threads[i].join();
        }
    }
}

```




## Deadlock

In general the following requirements for a deadlock can be identified:
* Mutual exclusion: There is a resource which can be accessed only by one thread at any point in time.
* Resource holding: While having locked one resource, the thread tries to acquire another lock on some other exclusive resource.
* No preemption: There is no mechanism, which frees the resource if one threads holds the lock for a specific period of time.
* Circular wait: During runtime a constellation occurs in which two (or more) threads are each waiting on the other thread to free a resource that it has locked.


* Monitor with wait and notify



```java

import java.util.Queue;
import java.util.concurrent.ConcurrentLinkedQueue;

public class SyncWaitNotfiyDemo {
    private static final Queue <Integer>queue = new ConcurrentLinkedQueue<Integer>();

    public Integer getNextInt() {
        Integer retVal = null;
        synchronized (queue) {
            try {
                while (queue.isEmpty()) {
                    queue.wait();
                }
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
            retVal = queue.poll();
        }
        return retVal;
    }

    public synchronized void putInt(Integer value) {
        synchronized (queue) {
            queue.add(value);
            queue.notify();
        }
    }

    public static void main(String[] args) throws InterruptedException {
        final SyncWaitNotfiyDemo queue = new SyncWaitNotfiyDemo();

        Thread thread1 = new Thread(new Runnable() {
            public void run() {
                for (int i = 0; i < 10; i++) {
                    queue.putInt(i);
                }
            }
        });

        Thread thread2 = new Thread(new Runnable() {
            public void run() {
                for (int i = 0; i < 10; i++) {
                    Integer nextInt = queue.getNextInt();
                    System.out.println("Next int: " + nextInt);
                }
            }
        });

        thread1.start();
        thread2.start();
        thread1.join();
        thread2.join();
    }
}
```

## Useful concurrent collections

### ConcurrentHashMap

> `ConcurrentHashMap` is undoubtedly most popular collection class introduced in Java 5 and most of us are already using it. It provides a concurrent alternative of Hashtable or Synchronized Map classes with aim to support higher level of concurrency by implementing fined grained locking. Multiple reader can access the Map concurrently  while a portion of Map gets locked for write operation depends upon concurrency level of Map. Also it provides better scalability than there synchronized counter part. Iterator of `ConcurrentHashMap` are fail-safe iterators which doesn't throw ConcurrencModificationException thus eliminates another requirement of locking during iteration which result in further scalability and performance.


### CopyOnWriteArrayList and CopyOnWriteArraySet

> `CopyOnWriteArrayList` is a concurrent alternative of synchronized List. It provides better concurrency than synchronized List by allowing multiple concurrent reader and replacing the whole list on write operation. Yes, write operation is costly on `CopyOnWriteArrayList` but it performs better when there are multiple reader and requirement of iteration is more than writing. Since `CopyOnWriteArrayList` Iterator also don't throw ConcurrencModificationException it eliminates need to lock the collection during iteration. Remember both `ConcurrentHashMap` and `CopyOnWriteArrayList` doesn't provides same level of locking as Synchronized Collection and achieves thread-safety by there locking and mutability strategy. So they perform better if requirements suits there nature. Similarly, `CopyOnWriteArraySet` is a concurrent replacement to Synchronized Set.

### BlockingQueue and `Deque`

> `BlockingQueue` makes it easy to implement producer-consumer design pattern by providing inbuilt blocking support for put() and take() method. put() method will block if Queue is full while take() method will block if Queue is empty. Java 5 API provides two concrete implementation of `BlockingQueue` in form of `ArrayBlockingQueue` and `LinkedBlockingQueue`, both of them implement FIFO ordering of element. ArrayBlockingQueue is backed by Array and its bounded in nature while `LinkedBlockingQueue` is optionally bounded. Consider using `BlockingQueue` to solve producer Consumer problem in Java instead of writing your won wait-notify code. Java 5 also provides `PriorityBlockingQueue`, another implementation of `BlockingQueue` which is ordered on priority and useful if you want to process elements on order other than FIFO.

> `Deque` interface is added in Java 6 and it extends Queue interface to support insertion and removal from both end of Queue referred as head and tail. Java6 also provides concurrent implementation of `Deque` like ArrayDeque and LinkedBlockingDeque. `Deque` Can be used efficiently to increase parallelism in program by allowing set of worker thread to help each other by taking some of work load from other thread by utilizing `Deque` double end consumption property. So if all Thread has there own set of task Queue and they are consuming from head; helper thread can also share some work load via consumption from tail.


### ConcurrentSkipListMap and ConcurrentSkipListSet

> Just like `ConcurrentHashMap` provides a concurrent alternative of synchronized HashMap. ConcurrentSkipListMap and ConcurrentSkipListSet provide concurrent alternative for synchronized version of SortedMap and SortedSet. For example instead of using TreeMap or TreeSet wrapped inside synchronized Collection, You can consider using ConcurrentSkipListMap or ConcurrentSkipListSet from java.util.concurrent package. They also implement NavigableMap and NavigableSet to add additional navigation method.


### Synchronizer

## Counter Semaphore

* Counting Semaphore in Java maintains specified number of pass or permits, In order to access a shared resource, Current Thread must acquire a permit. If permit is already exhausted by other thread than it can wait until a permit is available due to release of permit from different thread. This concurrency utility can be very useful to implement producer consumer design pattern or implement bounded pool or resources like Thread Pool, DB Connection pool etc.



```java

import java.util.concurrent.Semaphore;

public class SemaphoreDemo {

    Semaphore binary = new Semaphore(1);
  
    public static void main(String args[]) {
        final SemaphoreDemo test = new SemaphoreDemo();
        new Thread(){
            @Override
            public void run(){
              test.mutualExclusion(); 
            }
        }.start();
      
        new Thread(){
            @Override
            public void run(){
              test.mutualExclusion(); 
            }
        }.start();
      
    }
  
    private void mutualExclusion() {
        try {
            binary.acquire();

            //mutual exclusive region
            System.out.println(Thread.currentThread().getName() 
                + " inside mutual exclusive region");
            Thread.sleep(1000);

        } catch (InterruptedException i.e.) {
            ie.printStackTrace();
        } finally {
            binary.release();
            System.out.println(Thread.currentThread().getName() 
                + " outside of mutual exclusive region");
        }
    } 
  
}

// Output:
// Thread-0 inside mutual exclusive region
// Thread-0 outside of mutual exclusive region
// Thread-1 inside mutual exclusive region
// Thread-1 outside of mutual exclusive region

```


## CountDownLatch

* `CountDownLatch` in Java is a kind of synchronizer which allows one Thread  to wait for one or more Threads before starts processing. You can also implement same functionality using  wait and notify mechanism in Java but it requires lot of code and getting it write in first attempt is tricky,  With `CountDownLatch` it can  be done in just few lines. `CountDownLatch` also allows flexibility on number of thread for which main thread should wait, It can wait for one thread or n number of thread, there is not much change on code. 

* The difficulty to use it properly is where to use `CountDownLatch`. First, let us figour out how `CountDownLatch` works. usaullly main thread of application,  which calls CountDownLatch.await() will wait until count reaches zero or its interrupted by another Thread. All other thread are required to do count down by calling CountDownLatch.countDown() once they are completed. One disadvantage of `CountDownLatch` is not reusable, once its count reaches zero. 

* Sample program requires 3 services namely CacheService, AlertService  and ValidationService  to be started and ready before application can handle any request.

```java
import java.util.Date;
import java.util.concurrent.CountDownLatch;
import java.util.logging.Level;
import java.util.logging.Logger;

public class CountDownLatchDemo {

    public static void main(String args[]) {
       final CountDownLatch latch = new CountDownLatch(3);
       Thread cacheService = new Thread(new Service("CacheService", 1000, latch));
       Thread alertService = new Thread(new Service("AlertService", 1000, latch));
       Thread validationService = new Thread(new Service("ValidationService", 1000, latch));
      
       cacheService.start(); //separate thread will initialize CacheService
       alertService.start(); //another thread for AlertService initialization
       validationService.start();
      
      
       //count is 3 since we have 3 Threads (Services)
      
       try{
            latch.await();  //main thread is waiting on CountDownLatch to finish
            System.out.println("All services are up, Application is starting now");
       }catch(InterruptedException ie){
           ie.printStackTrace();
       }
      
    }
  
}

/**
 * Service class which will be executed by Thread using CountDownLatch synchronizer.
 */
class Service implements Runnable{
    private final String name;
    private final int timeToStart;
    private final CountDownLatch latch;
  
    public Service(String name, int timeToStart, CountDownLatch latch){
        this.name = name;
        this.timeToStart = timeToStart;
        this.latch = latch;
    }
  
    @Override
    public void run() {
        try {
            Thread.sleep(timeToStart);
        } catch (InterruptedException ex) {
            Logger.getLogger(Service.class.getName()).log(Level.SEVERE, null, ex);
        }
        System.out.println( name + " is Up");
        latch.countDown(); //reduce count of CountDownLatch by 1
    }  
}



```

## CylicBarrier

* `CyclicBarrier` is similar to `CountDownLatch` which we have seen in the last article  What is `CountDownLatch` in Java and allows multiple threads to wait for each other (barrier) before proceeding. The difference between `CountDownLatch` and `CyclicBarrier` is an also very popular multi-threading interview question in Java. `CyclicBarrier` is a natural requirement for a concurrent program because it can be used to perform final part of the task once individual tasks  are completed.

* The demo of `CyclicBarrier` on which we initialize `CyclicBarrier` with 3 parties, means in order to cross barrier, 3 thread needs to call await() method. each thread calls await method in short duration but they don't proceed until all 3 threads reached the barrier, once all thread reach the barrier, barrier gets broker and each thread started their execution from that point.

```java
import java.util.concurrent.BrokenBarrierException;
import java.util.concurrent.`CyclicBarrier`;
import java.util.logging.Level;
import java.util.logging.Logger;

public class CyclicBarrierDemo {

    //Runnable task for each thread
    private static class Task implements Runnable {

        private CyclicBarrier barrier;

        public Task(CyclicBarrier barrier) {
            this.barrier = barrier;
        }

        @Override
        public void run() {
            try {
                System.out.println(Thread.currentThread().getName() + " is waiting on barrier");
                barrier.await();
                System.out.println(Thread.currentThread().getName() + " has crossed the barrier");
            } catch (InterruptedException ex) {
                Logger.getLogger(CyclicBarrierDemo.class.getName()).log(Level.SEVERE, null, ex);
            } catch (BrokenBarrierException ex) {
                Logger.getLogger(CyclicBarrierDemo.class.getName()).log(Level.SEVERE, null, ex);
            }
        }
    }

    public static void main(String args[]) {

        //creating CyclicBarrier with 3 parties i.e. 3 Threads needs to call await()
        final CyclicBarrier cb = new CyclicBarrier(3, new Runnable(){
            @Override
            public void run(){
                //This task will be executed once all thread reaches barrier
                System.out.println("All parties are arrived at barrier, lets play");
            }
        });

        //starting each of thread
        Thread t1 = new Thread(new Task(cb), "Thread 1");
        Thread t2 = new Thread(new Task(cb), "Thread 2");
        Thread t3 = new Thread(new Task(cb), "Thread 3");

        t1.start();
        t2.start();
        t3.start();
      
    }
}

// Output:
// Thread 1 is waiting on barrier
// Thread 3 is waiting on barrier
// Thread 2 is waiting on barrier
// All parties have arrived at barrier, lets play
// Thread 3 has crossed the barrier
// Thread 1 has crossed the barrier
// Thread 2 has crossed the barrier

```


## Producer / Consumer pattern 

* Producer Consumer Design pattern is a classic concurrency or threading pattern which reduces coupling between
Producer and Consumer by separating Identification of work with Execution of Work. In producer consumer design pattern a shared queue is used to control the flow and this separation allows you to code producer and consumer separately.

* It is everywhere in real life and depict coordination and collaboration. Like one person is preparing food (Producer) while other one is serving food (Consumer), both will use shared table for putting food plates and taking food plates. Producer which is the person preparing food will wait if table is full and Consumer (Person who is serving food) will wait if table is empty. table is a shared object here. On Java library Executor framework itself implement Producer Consumer design pattern be separating responsibility of addition and execution of task.

```java
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.logging.Level;
import java.util.logging.Logger;

public class ProducerConsumerPattern {

    public static void main(String args[]){
  
     //Creating shared object
     BlockingQueue sharedQueue = new LinkedBlockingQueue();
 
     //Creating Producer and Consumer Thread
     Thread prodThread = new Thread(new Producer(sharedQueue));
     Thread consThread = new Thread(new Consumer(sharedQueue));

     //Starting producer and Consumer thread
     prodThread.start();
     consThread.start();
    }
 
}

//Producer Class in java
class Producer implements Runnable {

    private final BlockingQueue sharedQueue;

    public Producer(BlockingQueue sharedQueue) {
        this.sharedQueue = sharedQueue;
    }

    @Override
    public void run() {
        for(int i=0; i<10; i++){
            try {
                System.out.println("Produced: " + i);
                sharedQueue.put(i);
            } catch (InterruptedException ex) {
                Logger.getLogger(Producer.class.getName()).log(Level.SEVERE, null, ex);
            }
        }
    }

}

//Consumer Class in Java
class Consumer implements Runnable{

    private final BlockingQueue sharedQueue;

    public Consumer (BlockingQueue sharedQueue) {
        this.sharedQueue = sharedQueue;
    }
  
    @Override
    public void run() {
        while(true){
            try {
                System.out.println("Consumed: "+ sharedQueue.take());
            } catch (InterruptedException ex) {
                Logger.getLogger(Consumer.class.getName()).log(Level.SEVERE, null, ex);
            }
        }
    }
  
  
}

// Output:
// Produced: 0
// Produced: 1
// Consumed: 0
// Produced: 2
// Consumed: 1
// Produced: 3
// Consumed: 2
// Produced: 4
// Consumed: 3
// Produced: 5
// Consumed: 4
// Produced: 6
// Consumed: 5
// Produced: 7
// Consumed: 6
// Produced: 8
// Consumed: 7
// Produced: 9
// Consumed: 8
// Consumed: 9

```


## Executor -- Thread Pool

* Java 1.5 introduced Thread pool in Java in the form of Executor framework, which allows Java programmer to decouple submission of a task to execution of the task. It also introduced a full feature built-in Thread Pool framework commonly known as Executor framework. Executor framework also provides different kind of Thread Pool e.g. `SingleThreadExecutor` which creates just one worker thread or `CachedThreadPool` which creates worker threads as and when necessary. 

* Demo of thread pool

```java
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class ThreadPoolDemo {

    public static void main(String args[]) {
       ExecutorService service = Executors.newFixedThreadPool(10);
       for (int i =0; i<100; i++){
           service.submit(new Task(i));
       }
    }
  
}

final class Task implements Runnable{
    private int taskId;
  
    public Task(int id){
        this.taskId = id;
    }
  
    @Override
    public void run() {
        System.out.println("Task ID : " 
            + this.taskId +" performed by " 
            + Thread.currentThread().getName());
    }  
}

// Output:
// Task ID : 0 performed by pool-1-thread-1
// Task ID : 7 performed by pool-1-thread-8
// Task ID : 8 performed by pool-1-thread-9
// Task ID : 6 performed by pool-1-thread-7
// Task ID : 4 performed by pool-1-thread-5
// Task ID : 5 performed by pool-1-thread-6
// Task ID : 3 performed by pool-1-thread-4
// Task ID : 1 performed by pool-1-thread-2
// ...
```

## Submit(Runnable)

> The submit(Runnable) method also takes a Runnable implementation, but returns a Future object. This Future object can be used to check if the Runnable as finished executing.

## Submit(Callable)

> The submit(Callable) method is similar to the submit(Runnable) method except for the type of parameter it takes. The Callable instance is very similar to a Runnable except that its call() method can return a result.

## InvokeAny()

> The invokeAny() method takes a collection of Callable objects, or subinterfaces of Callable. If one of the tasks complete (or throws an exception), the rest of the Callable's are cancelled.

## InvokeAll()

> The invokeAll() method invokes all of the Callable objects and returns a list of Future objects. Keep in mind that a task might finish due to an exception, so it may not have "succeeded". There is no way on a Future to tell the difference.

## Demo of submit, InvokeAny

```java
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Callable;
import java.util.concurrent.Future;
import java.util.concurrent.ExecutionException;
import java.util.HashSet;
import java.util.Set;

public class SubmitInvokeDemo {

    public static void main(String args[]) throws InterruptedException, ExecutionException {


        ExecutorService executorService = Executors.newSingleThreadExecutor();

        Future future = executorService.submit(new Runnable() {
            public void run() {
                System.out.println("Asynchronous task");
            }
        });

        future.get();  //returns null if the task has finished correctly.

        Set<Callable<String>> callables = new HashSet<Callable<String>>();

        callables.add(new Callable<String>() {
            public String call() throws Exception {
                return "Task 1";
            }
        });
        callables.add(new Callable<String>() {
            public String call() throws Exception {
                return "Task 2";
            }
        });
        callables.add(new Callable<String>() {
            public String call() throws Exception {
                return "Task 3";
            }
        });

        String result = executorService.invokeAny(callables);

        System.out.println("result = " + result);

        executorService.shutdown();

    }
}

```




### Join and Fork 

* Here is an introduction into the Fork/Join Framework that is part of the JDK since version 1.7.

## Join and Fork with Executor Service

* The demo code submit() our tasks to the ExecutorService and then use the returned instance of `Future` to wait() for the result. The normal `ExecutorService` where you would have to block the current thread while waiting for a result. If we would only provide as many threads to the pool as we have CPUs available, the program would run out of resources and hang indefinitely.

```java
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Callable;
import java.util.concurrent.Future;
import java.util.concurrent.ExecutionException;
import java.util.Random;

public class FindMinTask implements Callable<Integer> {
    private int[] numbers;
    private int startIndex;
    private int endIndex;
    private ExecutorService executorService;

    public FindMinTask(
        ExecutorService executorService, 
        int[] numbers, int startIndex, int endIndex) {

        this.executorService = executorService;
        this.numbers = numbers;
        this.startIndex = startIndex;
        this.endIndex = endIndex;
    }

    public Integer call() throws Exception {

        int sliceLength = (endIndex - startIndex) + 1;
        if (sliceLength > 2) {
            FindMinTask lowerFindMin = new FindMinTask(
                    executorService, numbers, startIndex,
                    startIndex + (sliceLength / 2) - 1);
                    
            Future<Integer> futureLowerFindMin = executorService.submit(lowerFindMin);
            FindMinTask upperFindMin = new FindMinTask(
                    executorService, numbers, startIndex + (sliceLength / 2),
                    endIndex);

            Future<Integer> futureUpperFindMin = executorService.submit(upperFindMin);
            return Math.min(futureLowerFindMin.get(), futureUpperFindMin.get());
        } else {
            return Math.min(numbers[startIndex], numbers[endIndex]);
        }
    }

    public static void main(String[] args) 
        throws InterruptedException, ExecutionException {

        int[] numbers = new int[100];
        Random random = new Random(System.currentTimeMillis());
        for (int i = 0; i < numbers.length; i++) {
            numbers[i] = random.nextInt(100);
        }
        ExecutorService executorService = Executors.newFixedThreadPool(64);
        Future<Integer> futureResult = executorService
                .submit(new FindMinTask(
                        executorService, 
                        numbers, 0, numbers.length - 1));

        System.out.println(futureResult.get());
        executorService.shutdown();
    }
}
```


## Join and Fork with JoinForkPool

* The `ForkJoinPool` implements the already mentioned work-stealing strategy, i.e. every time a running thread has to wait for some result; the thread removes the current task from the work queue and executes some other task ready to run. This way the current thread is not blocked and can be used to execute other tasks. Once the result for the originally suspended task has been computed the task gets executed again and the join() method returns the result. This is an important difference between `JoinForkPool` and `ExecutorService`.

* Demo of JoinForkPool

```java
import java.awt.*;
import java.awt.image.*;
import java.io.*;
import java.util.concurrent.ForkJoinPool;
import javax.imageio.*;
import java.util.concurrent.RecursiveAction;

public class GrayscaleImageAction extends RecursiveAction {
    private static final long serialVersionUID = 1L;
    private int row;
    private BufferedImage bufferedImage;

    public GrayscaleImageAction(int row, BufferedImage bufferedImage) {
        this.row = row;
        this.bufferedImage = bufferedImage;
    }

    @Override
    protected void compute() {
        for (int column = 0; column < bufferedImage.getWidth(); column++) {
            int rgb = bufferedImage.getRGB(column, row);
            int r = (rgb >> 16) & 0xFF;
            int g = (rgb >> 8) & 0xFF;
            int b = (rgb & 0xFF);
            int gray = (int) (0.2126 * (float) r + 0.7152 * (float) g + 0.0722 * (float) b);
            gray = (gray << 16) + (gray << 8) + gray;
            bufferedImage.setRGB(column, row, gray);
        }
    }

    public static void main(String[] args) throws IOException {
        ForkJoinPool pool = new ForkJoinPool(
                Runtime.getRuntime().availableProcessors());
        BufferedImage bufferedImage = ImageIO.read(new File(args[0]));
        for (int row = 0; row < bufferedImage.getHeight(); row++) {
            GrayscaleImageAction action = new GrayscaleImageAction(
                row, bufferedImage);
            pool.execute(action);
        }
        pool.shutdown();
        ImageIO.write(bufferedImage, "jpg", new File(args[1]));
    }
}

```<file_sep>+++
title="Packaging"
description="Package, Visibility, Pitfalls"
weight=8
+++

### Package 


* Package is a way to structure code: a program is constructed as a “package” (often abbreviated as pkg), which may use facilities from other packages. Every go-file belongs to one (and only one) package (like a library or namespace in other languages). Many different .go files can belong to one package, so the filename(s) and package name are generally not the same.


* The package to which the code-file belongs must be indicated on the first line, e.g.: package main . A standalone executable belongs to package main. Each Go application contains one package called main. An application can consist of different packages, but even if you use only package main, you don’t have to stuff all code in 1 big file: you can make a number of smaller files each having package main as 1 st codeline. If you compile a source file with a package name other than main, like pack1, the object file is stored in pack1.a; a package name is written in lowercase letters.



#### Standard library

* The Go installation contains a number of ready-to-use packages, which form the standard library.

* To build a program, the packages, and the files within them, must be compiled in the correct order. Package dependencies determine the order in which to build packages.

#### Import

* A Go program is created by linking together a set of packages through the import keyword.


### VISIBILITY RULE

* When the identifier ( of a constant, variable, type, function, struct field, ...) starts with an uppercase letter, like Group1, then the ‘object’ with this identifier is visible in code outside the package (thus available to client-programs, ‘importers’ of the package), it is said to be exported (like public in OO languages). Identifiers which start with a lowercase letter are not visible outside the package, but they are visible and usable in the whole package (like private).

#### package alias 

* A package can, if this is useful (for shortening, name conflicts, ...), also be given another name (an alias), like: import fm “fmt” .

































<file_sep>+++

title = "C Lecture - 3"
description = "Exercise 41~ 48"
+++

Author: <NAME>

All content comes from Zed's [Lecture Repository](https://github.com/zedshaw/learn-c-the-hard-way-lectures.git) and [Libraries Repository](https://github.com/zedshaw/liblcthw).  All credit goes to Zed.



### Exercise 41 Project devpkg
.\ex41\devpkg

.\ex41\devpkg\commands.c

```c
#include <apr_uri.h>
#include <apr_fnmatch.h>
#include <unistd.h>

#include "commands.h"
#include "dbg.h"
#include "bstrlib.h"
#include "db.h"
#include "shell.h"

int Command_depends(apr_pool_t * p, const char *path)
{
    FILE *in = NULL;
    bstring line = NULL;

    in = fopen(path, "r");
    check(in != NULL, "Failed to open downloaded depends: %s", path);

    for (line = bgets((bNgetc) fgetc, in, '\n');
            line != NULL; 
            line = bgets((bNgetc) fgetc, in, '\n')) 
    {
        btrimws(line);
        log_info("Processing depends: %s", bdata(line));
        int rc = Command_install(p, bdata(line), NULL, NULL, NULL);
        check(rc == 0, "Failed to install: %s", bdata(line));
        bdestroy(line);
    }

    fclose(in);
    return 0;

error:
    if (line) bdestroy(line);
    if (in) fclose(in);
    return -1;
}

int Command_fetch(apr_pool_t * p, const char *url, int fetch_only)
{
    apr_uri_t info = {.port = 0 };
    int rc = 0;
    const char *depends_file = NULL;
    apr_status_t rv = apr_uri_parse(p, url, &info);

    check(rv == APR_SUCCESS, "Failed to parse URL: %s", url);

    if (apr_fnmatch(GIT_PAT, info.path, 0) == APR_SUCCESS) {
        rc = Shell_exec(GIT_SH, "URL", url, NULL);
        check(rc == 0, "git failed.");
    } else if (apr_fnmatch(DEPEND_PAT, info.path, 0) == APR_SUCCESS) {
        check(!fetch_only, "No point in fetching a DEPENDS file.");

        if (info.scheme) {
            depends_file = DEPENDS_PATH;
            rc = Shell_exec(CURL_SH, "URL", url, "TARGET", depends_file,
                    NULL);
            check(rc == 0, "Curl failed.");
        } else {
            depends_file = info.path;
        }

        // recursively process the devpkg list
        log_info("Building according to DEPENDS: %s", url);
        rv = Command_depends(p, depends_file);
        check(rv == 0, "Failed to process the DEPENDS: %s", url);

        // this indicates that nothing needs to be done
        return 0;

    } else if (apr_fnmatch(TAR_GZ_PAT, info.path, 0) == APR_SUCCESS) {
        if (info.scheme) {
            rc = Shell_exec(CURL_SH,
                    "URL", url, "TARGET", TAR_GZ_SRC, NULL);
            check(rc == 0, "Failed to curl source: %s", url);
        }

        rv = apr_dir_make_recursive(BUILD_DIR,
                APR_UREAD | APR_UWRITE |
                APR_UEXECUTE, p);
        check(rv == APR_SUCCESS, "Failed to make directory %s",
                BUILD_DIR);

        rc = Shell_exec(TAR_SH, "FILE", TAR_GZ_SRC, NULL);
        check(rc == 0, "Failed to untar %s", TAR_GZ_SRC);
    } else if (apr_fnmatch(TAR_BZ2_PAT, info.path, 0) == APR_SUCCESS) {
        if (info.scheme) {
            rc = Shell_exec(CURL_SH, "URL", url, "TARGET", TAR_BZ2_SRC,
                    NULL);
            check(rc == 0, "Curl failed.");
        }

        apr_status_t rc = apr_dir_make_recursive(BUILD_DIR,
                APR_UREAD | APR_UWRITE
                | APR_UEXECUTE, p);

        check(rc == 0, "Failed to make directory %s", BUILD_DIR);
        rc = Shell_exec(TAR_SH, "FILE", TAR_BZ2_SRC, NULL);
        check(rc == 0, "Failed to untar %s", TAR_BZ2_SRC);
    } else {
        sentinel("Don't now how to handle %s", url);
    }

    // indicates that an install needs to actually run
    return 1;
error:
    return -1;
}

int Command_build(apr_pool_t * p, const char *url,
        const char *configure_opts, const char *make_opts,
        const char *install_opts)
{
    int rc = 0;

    check(access(BUILD_DIR, X_OK | R_OK | W_OK) == 0,
            "Build directory doesn't exist: %s", BUILD_DIR);

    // actually do an install
    if (access(CONFIG_SCRIPT, X_OK) == 0) {
        log_info("Has a configure script, running it.");
        rc = Shell_exec(CONFIGURE_SH, "OPTS", configure_opts, NULL);
        check(rc == 0, "Failed to configure.");
    }

    rc = Shell_exec(MAKE_SH, "OPTS", make_opts, NULL);
    check(rc == 0, "Failed to build.");

    rc = Shell_exec(INSTALL_SH,
            "TARGET", install_opts ? install_opts : "install",
            NULL);
    check(rc == 0, "Failed to install.");

    rc = Shell_exec(CLEANUP_SH, NULL);
    check(rc == 0, "Failed to cleanup after build.");

    rc = DB_update(url);
    check(rc == 0, "Failed to add this package to the database.");

    return 0;

error:
    return -1;
}

int Command_install(apr_pool_t * p, const char *url,
        const char *configure_opts, const char *make_opts,
        const char *install_opts)
{
    int rc = 0;
    check(Shell_exec(CLEANUP_SH, NULL) == 0,
            "Failed to cleanup before building.");

    rc = DB_find(url);
    check(rc != -1, "Error checking the install database.");

    if (rc == 1) {
        log_info("Package %s already installed.", url);
        return 0;
    }

    rc = Command_fetch(p, url, 0);

    if (rc == 1) {
        rc = Command_build(p, url, configure_opts, make_opts,
                install_opts);
        check(rc == 0, "Failed to build: %s", url);
    } else if (rc == 0) {
        // no install needed
        log_info("Depends successfully installed: %s", url);
    } else {
        // had an error
        sentinel("Install failed: %s", url);
    }

    Shell_exec(CLEANUP_SH, NULL);
    return 0;

error:
    Shell_exec(CLEANUP_SH, NULL);
    return -1;
}

```

.\ex41\devpkg\commands.h

```c
#ifndef _commands_h
#define _commands_h

#include <apr_pools.h>

#define DEPENDS_PATH "/tmp/DEPENDS"
#define TAR_GZ_SRC "/tmp/pkg-src.tar.gz"
#define TAR_BZ2_SRC "/tmp/pkg-src.tar.bz2"
#define BUILD_DIR "/tmp/pkg-build"
#define GIT_PAT "*.git"
#define DEPEND_PAT "*DEPENDS"
#define TAR_GZ_PAT "*.tar.gz"
#define TAR_BZ2_PAT "*.tar.bz2"
#define CONFIG_SCRIPT "/tmp/pkg-build/configure"

enum CommandType {
    COMMAND_NONE, COMMAND_INSTALL, COMMAND_LIST, COMMAND_FETCH,
    COMMAND_INIT, COMMAND_BUILD
};

int Command_fetch(apr_pool_t * p, const char *url, int fetch_only);

int Command_install(apr_pool_t * p, const char *url,
        const char *configure_opts, const char *make_opts,
        const char *install_opts);

int Command_depends(apr_pool_t * p, const char *path);

int Command_build(apr_pool_t * p, const char *url,
        const char *configure_opts, const char *make_opts,
        const char *install_opts);

#endif

```

.\ex41\devpkg\db.c

```c
#include <unistd.h>
#include <apr_errno.h>
#include <apr_file_io.h>

#include "db.h"
#include "bstrlib.h"
#include "dbg.h"

static FILE *DB_open(const char *path, const char *mode)
{
    return fopen(path, mode);
}

static void DB_close(FILE * db)
{
    fclose(db);
}

static bstring DB_load()
{
    FILE *db = NULL;
    bstring data = NULL;

    db = DB_open(DB_FILE, "r");
    check(db, "Failed to open database: %s", DB_FILE);

    data = bread((bNread) fread, db);
    check(data, "Failed to read from db file: %s", DB_FILE);

    DB_close(db);
    return data;

error:
    if (db)
        DB_close(db);
    if (data)
        bdestroy(data);
    return NULL;
}

int DB_update(const char *url)
{
    if (DB_find(url)) {
        log_info("Already recorded as installed: %s", url);
    }

    FILE *db = DB_open(DB_FILE, "a+");
    check(db, "Failed to open DB file: %s", DB_FILE);

    bstring line = bfromcstr(url);
    bconchar(line, '\n');
    int rc = fwrite(line->data, blength(line), 1, db);
    check(rc == 1, "Failed to append to the db.");

    return 0;
error:
    if (db)
        DB_close(db);
    return -1;
}

int DB_find(const char *url)
{
    bstring data = NULL;
    bstring line = bfromcstr(url);
    int res = -1;

    data = DB_load();
    check(data, "Failed to load: %s", DB_FILE);

    if (binstr(data, 0, line) == BSTR_ERR) {
        res = 0;
    } else {
        res = 1;
    }

error:            // fallthrough
    if (data)
        bdestroy(data);
    if (line)
        bdestroy(line);

    return res;
}

int DB_init()
{
    apr_pool_t *p = NULL;
    apr_pool_initialize();
    apr_pool_create(&p, NULL);

    if (access(DB_DIR, W_OK | X_OK) == -1) {
        apr_status_t rc = apr_dir_make_recursive(DB_DIR,
                APR_UREAD | APR_UWRITE
                | APR_UEXECUTE |
                APR_GREAD | APR_GWRITE
                | APR_GEXECUTE, p);
        check(rc == APR_SUCCESS, "Failed to make database dir: %s",
                DB_DIR);
    }

    if (access(DB_FILE, W_OK) == -1) {
        FILE *db = DB_open(DB_FILE, "w");
        check(db, "Cannot open database: %s", DB_FILE);
        DB_close(db);
    }

    apr_pool_destroy(p);
    return 0;

error:
    apr_pool_destroy(p);
    return -1;
}

int DB_list()
{
    bstring data = DB_load();
    check(data, "Failed to read load: %s", DB_FILE);

    printf("%s", bdata(data));
    bdestroy(data);
    return 0;

error:
    return -1;
}

```

.\ex41\devpkg\db.h

```c
#ifndef _db_h
#define _db_h

#define DB_FILE "/usr/local/.devpkg/db"
#define DB_DIR "/usr/local/.devpkg"

int DB_init();
int DB_list();
int DB_update(const char *url);
int DB_find(const char *url);

#endif

```

.\ex41\devpkg\dbg.h

```c
#ifndef __dbg_h__
#define __dbg_h__

#include <stdio.h>
#include <errno.h>
#include <string.h>

#ifdef NDEBUG
#define debug(M, ...)
#else
#define debug(M, ...) fprintf(stderr, "DEBUG %s:%d: " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)
#endif

#define clean_errno() (errno == 0 ? "None" : strerror(errno))

#define log_err(M, ...) fprintf(stderr,\
        "[ERROR] (%s:%d: errno: %s) " M "\n", __FILE__, __LINE__,\
        clean_errno(), ##__VA_ARGS__)

#define log_warn(M, ...) fprintf(stderr,\
        "[WARN] (%s:%d: errno: %s) " M "\n",\
        __FILE__, __LINE__, clean_errno(), ##__VA_ARGS__)

#define log_info(M, ...) fprintf(stderr, "[INFO] (%s:%d) " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)

#define check(A, M, ...) if(!(A)) {\
    log_err(M, ##__VA_ARGS__); errno=0; goto error; }

#define sentinel(M, ...)  { log_err(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#define check_mem(A) check((A), "Out of memory.")

#define check_debug(A, M, ...) if(!(A)) { debug(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#endif

```

.\ex41\devpkg\devpkg.c

```c
#include <stdio.h>
#include <apr_general.h>
#include <apr_getopt.h>
#include <apr_strings.h>
#include <apr_lib.h>

#include "dbg.h"
#include "db.h"
#include "commands.h"

int main(int argc, const char *argv[])
{
    apr_pool_t *p = NULL;
    apr_pool_initialize();
    apr_pool_create(&p, NULL);

    apr_getopt_t *opt;
    apr_status_t rv;

    char ch = '\0';
    const char *optarg = NULL;
    const char *config_opts = NULL;
    const char *install_opts = NULL;
    const char *make_opts = NULL;
    const char *url = NULL;
    enum CommandType request = COMMAND_NONE;

    rv = apr_getopt_init(&opt, p, argc, argv);

    while (apr_getopt(opt, "I:Lc:m:i:d:SF:B:", &ch, &optarg) ==
            APR_SUCCESS) {
        switch (ch) {
            case 'I':
                request = COMMAND_INSTALL;
                url = optarg;
                break;

            case 'L':
                request = COMMAND_LIST;
                break;

            case 'c':
                config_opts = optarg;
                break;

            case 'm':
                make_opts = optarg;
                break;

            case 'i':
                install_opts = optarg;
                break;

            case 'S':
                request = COMMAND_INIT;
                break;

            case 'F':
                request = COMMAND_FETCH;
                url = optarg;
                break;

            case 'B':
                request = COMMAND_BUILD;
                url = optarg;
                break;
        }
    }

    switch (request) {
        case COMMAND_INSTALL:
            check(url, "You must at least give a URL.");
            Command_install(p, url, config_opts, make_opts, install_opts);
            break;

        case COMMAND_LIST:
            DB_list();
            break;

        case COMMAND_FETCH:
            check(url != NULL, "You must give a URL.");
            Command_fetch(p, url, 1);
            log_info("Downloaded to %s and in /tmp/", BUILD_DIR);
            break;

        case COMMAND_BUILD:
            check(url, "You must at least give a URL.");
            Command_build(p, url, config_opts, make_opts, install_opts);
            break;

        case COMMAND_INIT:
            rv = DB_init();
            check(rv == 0, "Failed to make the database.");
            break;

        default:
            sentinel("Invalid command given.");
    }

    return 0;

error:
    return 1;
}

```

.\ex41\devpkg\shell.c

```c
#include "shell.h"
#include "dbg.h"
#include <stdarg.h>

int Shell_exec(Shell template, ...)
{
    apr_pool_t *p = NULL;
    int rc = -1;
    apr_status_t rv = APR_SUCCESS;
    va_list argp;
    const char *key = NULL;
    const char *arg = NULL;
    int i = 0;

    rv = apr_pool_create(&p, NULL);
    check(rv == APR_SUCCESS, "Failed to create pool.");

    va_start(argp, template);

    for (key = va_arg(argp, const char *);
            key != NULL; key = va_arg(argp, const char *)) {
        arg = va_arg(argp, const char *);

        for (i = 0; template.args[i] != NULL; i++) {
            if (strcmp(template.args[i], key) == 0) {
                template.args[i] = arg;
                break;        // found it
            }
        }
    }

    rc = Shell_run(p, &template);
    apr_pool_destroy(p);
    va_end(argp);
    return rc;

error:
    if (p) {
        apr_pool_destroy(p);
    }
    return rc;
}

int Shell_run(apr_pool_t * p, Shell * cmd)
{
    apr_procattr_t *attr;
    apr_status_t rv;
    apr_proc_t newproc;

    rv = apr_procattr_create(&attr, p);
    check(rv == APR_SUCCESS, "Failed to create proc attr.");

    rv = apr_procattr_io_set(attr, APR_NO_PIPE, APR_NO_PIPE,
            APR_NO_PIPE);
    check(rv == APR_SUCCESS, "Failed to set IO of command.");

    rv = apr_procattr_dir_set(attr, cmd->dir);
    check(rv == APR_SUCCESS, "Failed to set root to %s", cmd->dir);

    rv = apr_procattr_cmdtype_set(attr, APR_PROGRAM_PATH);
    check(rv == APR_SUCCESS, "Failed to set cmd type.");

    rv = apr_proc_create(&newproc, cmd->exe, cmd->args, NULL, attr, p);
    check(rv == APR_SUCCESS, "Failed to run command.");

    rv = apr_proc_wait(&newproc, &cmd->exit_code, &cmd->exit_why,
            APR_WAIT);
    check(rv == APR_CHILD_DONE, "Failed to wait.");

    check(cmd->exit_code == 0, "%s exited badly.", cmd->exe);
    check(cmd->exit_why == APR_PROC_EXIT, "%s was killed or crashed",
            cmd->exe);

    return 0;

error:
    return -1;
}

Shell CLEANUP_SH = {
    .exe = "rm",
    .dir = "/tmp",
    .args = {"rm", "-rf", "/tmp/pkg-build", "/tmp/pkg-src.tar.gz",
        "/tmp/pkg-src.tar.bz2", "/tmp/DEPENDS", NULL}
};

Shell GIT_SH = {
    .dir = "/tmp",
    .exe = "git",
    .args = {"git", "clone", "URL", "pkg-build", NULL}
};

Shell TAR_SH = {
    .dir = "/tmp/pkg-build",
    .exe = "tar",
    .args = {"tar", "-xzf", "FILE", "--strip-components", "1", NULL}
};

Shell CURL_SH = {
    .dir = "/tmp",
    .exe = "curl",
    .args = {"curl", "-L", "-o", "TARGET", "URL", NULL}
};

Shell CONFIGURE_SH = {
    .exe = "./configure",
    .dir = "/tmp/pkg-build",
    .args = {"configure", "OPTS", NULL}
    ,
};

Shell MAKE_SH = {
    .exe = "make",
    .dir = "/tmp/pkg-build",
    .args = {"make", "OPTS", NULL}
};

Shell INSTALL_SH = {
    .exe = "sudo",
    .dir = "/tmp/pkg-build",
    .args = {"sudo", "make", "TARGET", NULL}
};

```

.\ex41\devpkg\shell.h

```c
#ifndef _shell_h
#define _shell_h

#define MAX_COMMAND_ARGS 100

#include <apr_thread_proc.h>

typedef struct Shell {
    const char *dir;
    const char *exe;

    apr_procattr_t *attr;
    apr_proc_t proc;
    apr_exit_why_e exit_why;
    int exit_code;

    const char *args[MAX_COMMAND_ARGS];
} Shell;

int Shell_run(apr_pool_t * p, Shell * cmd);
int Shell_exec(Shell cmd, ...);

extern Shell CLEANUP_SH;
extern Shell GIT_SH;
extern Shell TAR_SH;
extern Shell CURL_SH;
extern Shell CONFIGURE_SH;
extern Shell MAKE_SH;
extern Shell INSTALL_SH;

#endif

```

.\ex41\ex41.1.sh

```bash
set -e

## go somewhere safe
cd /tmp

## get the source to base APR 1.5.2
curl -L -O http://archive.apache.org/dist/apr/apr-1.5.2.tar.gz

## extract it and go into the source
tar -xzvf apr-1.5.2.tar.gz
cd apr-1.5.2

## you need this on OSX Yosemite
touch libtoolT

## configure, make, make install
./configure
make
sudo make install

## reset and cleanup
cd /tmp
rm -rf apr-1.5.2 apr-1.5.2.tar.gz

## do the same with apr-util
curl -L -O http://archive.apache.org/dist/apr/apr-util-1.5.4.tar.gz

## extract
tar -xzvf apr-util-1.5.4.tar.gz
cd apr-util-1.5.4

## you need this on OSX Yosemite
touch libtoolT

## configure, make, make install
./configure --with-apr=/usr/local/apr
## you need that extra parameter to configure because
## apr-util can't really find it because...who knows.

make
sudo make install

#cleanup
cd /tmp
rm -rf apr-util-1.5.4* apr-1.5.2*

```

.\ex41\ex41.2.sh

```bash
mkdir devpkg
cd devpkg
touch README Makefile

```

.\ex41\devpkg

.\ex41\devpkg\commands.c

```c
#include <apr_uri.h>
#include <apr_fnmatch.h>
#include <unistd.h>

#include "commands.h"
#include "dbg.h"
#include "bstrlib.h"
#include "db.h"
#include "shell.h"

int Command_depends(apr_pool_t * p, const char *path)
{
    FILE *in = NULL;
    bstring line = NULL;

    in = fopen(path, "r");
    check(in != NULL, "Failed to open downloaded depends: %s", path);

    for (line = bgets((bNgetc) fgetc, in, '\n');
            line != NULL; 
            line = bgets((bNgetc) fgetc, in, '\n')) 
    {
        btrimws(line);
        log_info("Processing depends: %s", bdata(line));
        int rc = Command_install(p, bdata(line), NULL, NULL, NULL);
        check(rc == 0, "Failed to install: %s", bdata(line));
        bdestroy(line);
    }

    fclose(in);
    return 0;

error:
    if (line) bdestroy(line);
    if (in) fclose(in);
    return -1;
}

int Command_fetch(apr_pool_t * p, const char *url, int fetch_only)
{
    apr_uri_t info = {.port = 0 };
    int rc = 0;
    const char *depends_file = NULL;
    apr_status_t rv = apr_uri_parse(p, url, &info);

    check(rv == APR_SUCCESS, "Failed to parse URL: %s", url);

    if (apr_fnmatch(GIT_PAT, info.path, 0) == APR_SUCCESS) {
        rc = Shell_exec(GIT_SH, "URL", url, NULL);
        check(rc == 0, "git failed.");
    } else if (apr_fnmatch(DEPEND_PAT, info.path, 0) == APR_SUCCESS) {
        check(!fetch_only, "No point in fetching a DEPENDS file.");

        if (info.scheme) {
            depends_file = DEPENDS_PATH;
            rc = Shell_exec(CURL_SH, "URL", url, "TARGET", depends_file,
                    NULL);
            check(rc == 0, "Curl failed.");
        } else {
            depends_file = info.path;
        }

        // recursively process the devpkg list
        log_info("Building according to DEPENDS: %s", url);
        rv = Command_depends(p, depends_file);
        check(rv == 0, "Failed to process the DEPENDS: %s", url);

        // this indicates that nothing needs to be done
        return 0;

    } else if (apr_fnmatch(TAR_GZ_PAT, info.path, 0) == APR_SUCCESS) {
        if (info.scheme) {
            rc = Shell_exec(CURL_SH,
                    "URL", url, "TARGET", TAR_GZ_SRC, NULL);
            check(rc == 0, "Failed to curl source: %s", url);
        }

        rv = apr_dir_make_recursive(BUILD_DIR,
                APR_UREAD | APR_UWRITE |
                APR_UEXECUTE, p);
        check(rv == APR_SUCCESS, "Failed to make directory %s",
                BUILD_DIR);

        rc = Shell_exec(TAR_SH, "FILE", TAR_GZ_SRC, NULL);
        check(rc == 0, "Failed to untar %s", TAR_GZ_SRC);
    } else if (apr_fnmatch(TAR_BZ2_PAT, info.path, 0) == APR_SUCCESS) {
        if (info.scheme) {
            rc = Shell_exec(CURL_SH, "URL", url, "TARGET", TAR_BZ2_SRC,
                    NULL);
            check(rc == 0, "Curl failed.");
        }

        apr_status_t rc = apr_dir_make_recursive(BUILD_DIR,
                APR_UREAD | APR_UWRITE
                | APR_UEXECUTE, p);

        check(rc == 0, "Failed to make directory %s", BUILD_DIR);
        rc = Shell_exec(TAR_SH, "FILE", TAR_BZ2_SRC, NULL);
        check(rc == 0, "Failed to untar %s", TAR_BZ2_SRC);
    } else {
        sentinel("Don't now how to handle %s", url);
    }

    // indicates that an install needs to actually run
    return 1;
error:
    return -1;
}

int Command_build(apr_pool_t * p, const char *url,
        const char *configure_opts, const char *make_opts,
        const char *install_opts)
{
    int rc = 0;

    check(access(BUILD_DIR, X_OK | R_OK | W_OK) == 0,
            "Build directory doesn't exist: %s", BUILD_DIR);

    // actually do an install
    if (access(CONFIG_SCRIPT, X_OK) == 0) {
        log_info("Has a configure script, running it.");
        rc = Shell_exec(CONFIGURE_SH, "OPTS", configure_opts, NULL);
        check(rc == 0, "Failed to configure.");
    }

    rc = Shell_exec(MAKE_SH, "OPTS", make_opts, NULL);
    check(rc == 0, "Failed to build.");

    rc = Shell_exec(INSTALL_SH,
            "TARGET", install_opts ? install_opts : "install",
            NULL);
    check(rc == 0, "Failed to install.");

    rc = Shell_exec(CLEANUP_SH, NULL);
    check(rc == 0, "Failed to cleanup after build.");

    rc = DB_update(url);
    check(rc == 0, "Failed to add this package to the database.");

    return 0;

error:
    return -1;
}

int Command_install(apr_pool_t * p, const char *url,
        const char *configure_opts, const char *make_opts,
        const char *install_opts)
{
    int rc = 0;
    check(Shell_exec(CLEANUP_SH, NULL) == 0,
            "Failed to cleanup before building.");

    rc = DB_find(url);
    check(rc != -1, "Error checking the install database.");

    if (rc == 1) {
        log_info("Package %s already installed.", url);
        return 0;
    }

    rc = Command_fetch(p, url, 0);

    if (rc == 1) {
        rc = Command_build(p, url, configure_opts, make_opts,
                install_opts);
        check(rc == 0, "Failed to build: %s", url);
    } else if (rc == 0) {
        // no install needed
        log_info("Depends successfully installed: %s", url);
    } else {
        // had an error
        sentinel("Install failed: %s", url);
    }

    Shell_exec(CLEANUP_SH, NULL);
    return 0;

error:
    Shell_exec(CLEANUP_SH, NULL);
    return -1;
}

```

.\ex41\devpkg\commands.h

```c
#ifndef _commands_h
#define _commands_h

#include <apr_pools.h>

#define DEPENDS_PATH "/tmp/DEPENDS"
#define TAR_GZ_SRC "/tmp/pkg-src.tar.gz"
#define TAR_BZ2_SRC "/tmp/pkg-src.tar.bz2"
#define BUILD_DIR "/tmp/pkg-build"
#define GIT_PAT "*.git"
#define DEPEND_PAT "*DEPENDS"
#define TAR_GZ_PAT "*.tar.gz"
#define TAR_BZ2_PAT "*.tar.bz2"
#define CONFIG_SCRIPT "/tmp/pkg-build/configure"

enum CommandType {
    COMMAND_NONE, COMMAND_INSTALL, COMMAND_LIST, COMMAND_FETCH,
    COMMAND_INIT, COMMAND_BUILD
};

int Command_fetch(apr_pool_t * p, const char *url, int fetch_only);

int Command_install(apr_pool_t * p, const char *url,
        const char *configure_opts, const char *make_opts,
        const char *install_opts);

int Command_depends(apr_pool_t * p, const char *path);

int Command_build(apr_pool_t * p, const char *url,
        const char *configure_opts, const char *make_opts,
        const char *install_opts);

#endif

```

.\ex41\devpkg\db.c

```c
#include <unistd.h>
#include <apr_errno.h>
#include <apr_file_io.h>

#include "db.h"
#include "bstrlib.h"
#include "dbg.h"

static FILE *DB_open(const char *path, const char *mode)
{
    return fopen(path, mode);
}

static void DB_close(FILE * db)
{
    fclose(db);
}

static bstring DB_load()
{
    FILE *db = NULL;
    bstring data = NULL;

    db = DB_open(DB_FILE, "r");
    check(db, "Failed to open database: %s", DB_FILE);

    data = bread((bNread) fread, db);
    check(data, "Failed to read from db file: %s", DB_FILE);

    DB_close(db);
    return data;

error:
    if (db)
        DB_close(db);
    if (data)
        bdestroy(data);
    return NULL;
}

int DB_update(const char *url)
{
    if (DB_find(url)) {
        log_info("Already recorded as installed: %s", url);
    }

    FILE *db = DB_open(DB_FILE, "a+");
    check(db, "Failed to open DB file: %s", DB_FILE);

    bstring line = bfromcstr(url);
    bconchar(line, '\n');
    int rc = fwrite(line->data, blength(line), 1, db);
    check(rc == 1, "Failed to append to the db.");

    return 0;
error:
    if (db)
        DB_close(db);
    return -1;
}

int DB_find(const char *url)
{
    bstring data = NULL;
    bstring line = bfromcstr(url);
    int res = -1;

    data = DB_load();
    check(data, "Failed to load: %s", DB_FILE);

    if (binstr(data, 0, line) == BSTR_ERR) {
        res = 0;
    } else {
        res = 1;
    }

error:            // fallthrough
    if (data)
        bdestroy(data);
    if (line)
        bdestroy(line);

    return res;
}

int DB_init()
{
    apr_pool_t *p = NULL;
    apr_pool_initialize();
    apr_pool_create(&p, NULL);

    if (access(DB_DIR, W_OK | X_OK) == -1) {
        apr_status_t rc = apr_dir_make_recursive(DB_DIR,
                APR_UREAD | APR_UWRITE
                | APR_UEXECUTE |
                APR_GREAD | APR_GWRITE
                | APR_GEXECUTE, p);
        check(rc == APR_SUCCESS, "Failed to make database dir: %s",
                DB_DIR);
    }

    if (access(DB_FILE, W_OK) == -1) {
        FILE *db = DB_open(DB_FILE, "w");
        check(db, "Cannot open database: %s", DB_FILE);
        DB_close(db);
    }

    apr_pool_destroy(p);
    return 0;

error:
    apr_pool_destroy(p);
    return -1;
}

int DB_list()
{
    bstring data = DB_load();
    check(data, "Failed to read load: %s", DB_FILE);

    printf("%s", bdata(data));
    bdestroy(data);
    return 0;

error:
    return -1;
}

```

.\ex41\devpkg\db.h

```c
#ifndef _db_h
#define _db_h

#define DB_FILE "/usr/local/.devpkg/db"
#define DB_DIR "/usr/local/.devpkg"

int DB_init();
int DB_list();
int DB_update(const char *url);
int DB_find(const char *url);

#endif

```

.\ex41\devpkg\dbg.h

```c
#ifndef __dbg_h__
#define __dbg_h__

#include <stdio.h>
#include <errno.h>
#include <string.h>

#ifdef NDEBUG
#define debug(M, ...)
#else
#define debug(M, ...) fprintf(stderr, "DEBUG %s:%d: " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)
#endif

#define clean_errno() (errno == 0 ? "None" : strerror(errno))

#define log_err(M, ...) fprintf(stderr,\
        "[ERROR] (%s:%d: errno: %s) " M "\n", __FILE__, __LINE__,\
        clean_errno(), ##__VA_ARGS__)

#define log_warn(M, ...) fprintf(stderr,\
        "[WARN] (%s:%d: errno: %s) " M "\n",\
        __FILE__, __LINE__, clean_errno(), ##__VA_ARGS__)

#define log_info(M, ...) fprintf(stderr, "[INFO] (%s:%d) " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)

#define check(A, M, ...) if(!(A)) {\
    log_err(M, ##__VA_ARGS__); errno=0; goto error; }

#define sentinel(M, ...)  { log_err(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#define check_mem(A) check((A), "Out of memory.")

#define check_debug(A, M, ...) if(!(A)) { debug(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#endif

```

.\ex41\devpkg\devpkg.c

```c
#include <stdio.h>
#include <apr_general.h>
#include <apr_getopt.h>
#include <apr_strings.h>
#include <apr_lib.h>

#include "dbg.h"
#include "db.h"
#include "commands.h"

int main(int argc, const char *argv[])
{
    apr_pool_t *p = NULL;
    apr_pool_initialize();
    apr_pool_create(&p, NULL);

    apr_getopt_t *opt;
    apr_status_t rv;

    char ch = '\0';
    const char *optarg = NULL;
    const char *config_opts = NULL;
    const char *install_opts = NULL;
    const char *make_opts = NULL;
    const char *url = NULL;
    enum CommandType request = COMMAND_NONE;

    rv = apr_getopt_init(&opt, p, argc, argv);

    while (apr_getopt(opt, "I:Lc:m:i:d:SF:B:", &ch, &optarg) ==
            APR_SUCCESS) {
        switch (ch) {
            case 'I':
                request = COMMAND_INSTALL;
                url = optarg;
                break;

            case 'L':
                request = COMMAND_LIST;
                break;

            case 'c':
                config_opts = optarg;
                break;

            case 'm':
                make_opts = optarg;
                break;

            case 'i':
                install_opts = optarg;
                break;

            case 'S':
                request = COMMAND_INIT;
                break;

            case 'F':
                request = COMMAND_FETCH;
                url = optarg;
                break;

            case 'B':
                request = COMMAND_BUILD;
                url = optarg;
                break;
        }
    }

    switch (request) {
        case COMMAND_INSTALL:
            check(url, "You must at least give a URL.");
            Command_install(p, url, config_opts, make_opts, install_opts);
            break;

        case COMMAND_LIST:
            DB_list();
            break;

        case COMMAND_FETCH:
            check(url != NULL, "You must give a URL.");
            Command_fetch(p, url, 1);
            log_info("Downloaded to %s and in /tmp/", BUILD_DIR);
            break;

        case COMMAND_BUILD:
            check(url, "You must at least give a URL.");
            Command_build(p, url, config_opts, make_opts, install_opts);
            break;

        case COMMAND_INIT:
            rv = DB_init();
            check(rv == 0, "Failed to make the database.");
            break;

        default:
            sentinel("Invalid command given.");
    }

    return 0;

error:
    return 1;
}

```

.\ex41\devpkg\shell.c

```c
#include "shell.h"
#include "dbg.h"
#include <stdarg.h>

int Shell_exec(Shell template, ...)
{
    apr_pool_t *p = NULL;
    int rc = -1;
    apr_status_t rv = APR_SUCCESS;
    va_list argp;
    const char *key = NULL;
    const char *arg = NULL;
    int i = 0;

    rv = apr_pool_create(&p, NULL);
    check(rv == APR_SUCCESS, "Failed to create pool.");

    va_start(argp, template);

    for (key = va_arg(argp, const char *);
            key != NULL; key = va_arg(argp, const char *)) {
        arg = va_arg(argp, const char *);

        for (i = 0; template.args[i] != NULL; i++) {
            if (strcmp(template.args[i], key) == 0) {
                template.args[i] = arg;
                break;        // found it
            }
        }
    }

    rc = Shell_run(p, &template);
    apr_pool_destroy(p);
    va_end(argp);
    return rc;

error:
    if (p) {
        apr_pool_destroy(p);
    }
    return rc;
}

int Shell_run(apr_pool_t * p, Shell * cmd)
{
    apr_procattr_t *attr;
    apr_status_t rv;
    apr_proc_t newproc;

    rv = apr_procattr_create(&attr, p);
    check(rv == APR_SUCCESS, "Failed to create proc attr.");

    rv = apr_procattr_io_set(attr, APR_NO_PIPE, APR_NO_PIPE,
            APR_NO_PIPE);
    check(rv == APR_SUCCESS, "Failed to set IO of command.");

    rv = apr_procattr_dir_set(attr, cmd->dir);
    check(rv == APR_SUCCESS, "Failed to set root to %s", cmd->dir);

    rv = apr_procattr_cmdtype_set(attr, APR_PROGRAM_PATH);
    check(rv == APR_SUCCESS, "Failed to set cmd type.");

    rv = apr_proc_create(&newproc, cmd->exe, cmd->args, NULL, attr, p);
    check(rv == APR_SUCCESS, "Failed to run command.");

    rv = apr_proc_wait(&newproc, &cmd->exit_code, &cmd->exit_why,
            APR_WAIT);
    check(rv == APR_CHILD_DONE, "Failed to wait.");

    check(cmd->exit_code == 0, "%s exited badly.", cmd->exe);
    check(cmd->exit_why == APR_PROC_EXIT, "%s was killed or crashed",
            cmd->exe);

    return 0;

error:
    return -1;
}

Shell CLEANUP_SH = {
    .exe = "rm",
    .dir = "/tmp",
    .args = {"rm", "-rf", "/tmp/pkg-build", "/tmp/pkg-src.tar.gz",
        "/tmp/pkg-src.tar.bz2", "/tmp/DEPENDS", NULL}
};

Shell GIT_SH = {
    .dir = "/tmp",
    .exe = "git",
    .args = {"git", "clone", "URL", "pkg-build", NULL}
};

Shell TAR_SH = {
    .dir = "/tmp/pkg-build",
    .exe = "tar",
    .args = {"tar", "-xzf", "FILE", "--strip-components", "1", NULL}
};

Shell CURL_SH = {
    .dir = "/tmp",
    .exe = "curl",
    .args = {"curl", "-L", "-o", "TARGET", "URL", NULL}
};

Shell CONFIGURE_SH = {
    .exe = "./configure",
    .dir = "/tmp/pkg-build",
    .args = {"configure", "OPTS", NULL}
    ,
};

Shell MAKE_SH = {
    .exe = "make",
    .dir = "/tmp/pkg-build",
    .args = {"make", "OPTS", NULL}
};

Shell INSTALL_SH = {
    .exe = "sudo",
    .dir = "/tmp/pkg-build",
    .args = {"sudo", "make", "TARGET", NULL}
};

```

.\ex41\devpkg\shell.h

```c
#ifndef _shell_h
#define _shell_h

#define MAX_COMMAND_ARGS 100

#include <apr_thread_proc.h>

typedef struct Shell {
    const char *dir;
    const char *exe;

    apr_procattr_t *attr;
    apr_proc_t proc;
    apr_exit_why_e exit_why;
    int exit_code;

    const char *args[MAX_COMMAND_ARGS];
} Shell;

int Shell_run(apr_pool_t * p, Shell * cmd);
int Shell_exec(Shell cmd, ...);

extern Shell CLEANUP_SH;
extern Shell GIT_SH;
extern Shell TAR_SH;
extern Shell CURL_SH;
extern Shell CONFIGURE_SH;
extern Shell MAKE_SH;
extern Shell INSTALL_SH;

#endif

```

.\ex41\ex41.1.sh

```bash
set -e

## go somewhere safe
cd /tmp

## get the source to base APR 1.5.2
curl -L -O http://archive.apache.org/dist/apr/apr-1.5.2.tar.gz

## extract it and go into the source
tar -xzvf apr-1.5.2.tar.gz
cd apr-1.5.2

## you need this on OSX Yosemite
touch libtoolT

## configure, make, make install
./configure
make
sudo make install

## reset and cleanup
cd /tmp
rm -rf apr-1.5.2 apr-1.5.2.tar.gz

## do the same with apr-util
curl -L -O http://archive.apache.org/dist/apr/apr-util-1.5.4.tar.gz

## extract
tar -xzvf apr-util-1.5.4.tar.gz
cd apr-util-1.5.4

## you need this on OSX Yosemite
touch libtoolT

## configure, make, make install
./configure --with-apr=/usr/local/apr
## you need that extra parameter to configure because
## apr-util can't really find it because...who knows.

make
sudo make install

#cleanup
cd /tmp
rm -rf apr-util-1.5.4* apr-1.5.2*

```

.\ex41\ex41.2.sh

```bash
mkdir devpkg
cd devpkg
touch README Makefile

```

The Plan

Create a handy little tool called devpkg.

This will be a *lot* of work, so this video is more complete.

Demonstration

I'll demonstrate how devpkg works so you get a better idea.

Read the book's description as well for more details.

The Apache Portable Runtime

Review of the APR and installing it.

The Analysis

Walk through the code, where everything is, and what to watch out for.

Getting My Code

If you get stuck you can check out the learn-c-the-hard-way-lectures project:

And look in ex41/devpkg for the code.

Extra Credit

* Compare your code to my code available online.  Starting with 100%,
  remove 1% for each line you got wrong.
* Take the notes.txt file that you previously created and implement your improvements to the the code and functionality
  of ``devpkg``.
* Write an alternative version of ``devpkg`` using your other
  favorite language or the one you think can do this the best.  Compare
  the two, then improve your *C* version of ``devpkg`` based on what
  you've learned.

### Exercise 42 Stacks and Queues

The Plan

Create a Stack and Queue data structure from just the unit tests.

PAUSE!

WARNING!  Stop the video now and try to solve this yourself!

I'll show you how I did it after you try it (or you can cheat).

Code Review

.\ex42\queue_tests.c

```c
#include "minunit.h"
#include <lcthw/queue.h>
#include <assert.h>

static Queue *queue = NULL;
char *tests[] = { "test1 data", "test2 data", "test3 data" };

#define NUM_TESTS 3

char *test_create()
{
    queue = Queue_create();
    mu_assert(queue != NULL, "Failed to create queue.");

    return NULL;
}

char *test_destroy()
{
    mu_assert(queue != NULL, "Failed to make queue #2");
    Queue_destroy(queue);

    return NULL;
}

char *test_send_recv()
{
    int i = 0;
    for (i = 0; i < NUM_TESTS; i++) {
        Queue_send(queue, tests[i]);
        mu_assert(Queue_peek(queue) == tests[0], "Wrong next value.");
    }

    mu_assert(Queue_count(queue) == NUM_TESTS, "Wrong count on send.");

    QUEUE_FOREACH(queue, cur) {
        debug("VAL: %s", (char *)cur->value);
    }

    for (i = 0; i < NUM_TESTS; i++) {
        char *val = Queue_recv(queue);
        mu_assert(val == tests[i], "Wrong value on recv.");
    }

    mu_assert(Queue_count(queue) == 0, "Wrong count after recv.");

    return NULL;
}

char *all_tests()
{
    mu_suite_start();

    mu_run_test(test_create);
    mu_run_test(test_send_recv);
    mu_run_test(test_destroy);

    return NULL;
}

RUN_TESTS(all_tests);

```

.\ex42\stack_tests.c

```c
#include "minunit.h"
#include <lcthw/stack.h>
#include <assert.h>

static Stack *stack = NULL;
char *tests[] = { "test1 data", "test2 data", "test3 data" };

#define NUM_TESTS 3

char *test_create()
{
    stack = Stack_create();
    mu_assert(stack != NULL, "Failed to create stack.");

    return NULL;
}

char *test_destroy()
{
    mu_assert(stack != NULL, "Failed to make stack #2");
    Stack_destroy(stack);

    return NULL;
}

char *test_push_pop()
{
    int i = 0;
    for (i = 0; i < NUM_TESTS; i++) {
        Stack_push(stack, tests[i]);
        mu_assert(Stack_peek(stack) == tests[i], "Wrong next value.");
    }

    mu_assert(Stack_count(stack) == NUM_TESTS, "Wrong count on push.");

    STACK_FOREACH(stack, cur) {
        debug("VAL: %s", (char *)cur->value);
    }

    for (i = NUM_TESTS - 1; i >= 0; i--) {
        char *val = Stack_pop(stack);
        mu_assert(val == tests[i], "Wrong value on pop.");
    }

    mu_assert(Stack_count(stack) == 0, "Wrong count after pop.");

    return NULL;
}

char *all_tests()
{
    mu_suite_start();

    mu_run_test(test_create);
    mu_run_test(test_push_pop);
    mu_run_test(test_destroy);

    return NULL;
}

RUN_TESTS(all_tests);

```

Extra Credit

* Implement ``Stack`` using ``DArray`` instead of ``List``, but without changing the unit test.  That means you'll have to create your own ``STACK_FOREACH``.

### Exercise 43 A Simple Statistics Engine
.\ex43\stats.h

```c
#ifndef lcthw_stats_h
#define lcthw_stats_h

typedef struct Stats {
    double sum;
    double sumsq;
    unsigned long n;
    double min;
    double max;
} Stats;

Stats *Stats_recreate(double sum, double sumsq, unsigned long n,
        double min, double max);

Stats *Stats_create();

double Stats_mean(Stats * st);

double Stats_stddev(Stats * st);

void Stats_sample(Stats * st, double s);

void Stats_dump(Stats * st);

#endif

```

.\ex43\stats.c

```c
#include <math.h>
#include <lcthw/stats.h>
#include <stdlib.h>
#include <lcthw/dbg.h>

Stats *Stats_recreate(double sum, double sumsq, unsigned long n,
        double min, double max)
{
    Stats *st = malloc(sizeof(Stats));
    check_mem(st);

    st->sum = sum;
    st->sumsq = sumsq;
    st->n = n;
    st->min = min;
    st->max = max;

    return st;

error:
    return NULL;
}

Stats *Stats_create()
{
    return Stats_recreate(0.0, 0.0, 0L, 0.0, 0.0);
}

double Stats_mean(Stats * st)
{
    return st->sum / st->n;
}

double Stats_stddev(Stats * st)
{
    return sqrt((st->sumsq - (st->sum * st->sum / st->n)) /
            (st->n - 1));
}

void Stats_sample(Stats * st, double s)
{
    st->sum += s;
    st->sumsq += s * s;

    if (st->n == 0) {
        st->min = s;
        st->max = s;
    } else {
        if (st->min > s)
            st->min = s;
        if (st->max < s)
            st->max = s;
    }

    st->n += 1;
}

void Stats_dump(Stats * st)
{
    fprintf(stderr,
            "sum: %f, sumsq: %f, n: %ld, "
            "min: %f, max: %f, mean: %f, stddev: %f",
            st->sum, st->sumsq, st->n, st->min, st->max, Stats_mean(st),
            Stats_stddev(st));
}

```

.\ex43\stats_tests.c

```c
#include "minunit.h"
#include <lcthw/stats.h>
#include <math.h>

const int NUM_SAMPLES = 10;
double samples[] = {
    6.1061334, 9.6783204, 1.2747090, 8.2395131, 0.3333483,
    6.9755066, 1.0626275, 7.6587523, 4.9382973, 9.5788115
};

Stats expect = {
    .sumsq = 425.1641,
    .sum = 55.84602,
    .min = 0.333,
    .max = 9.678,
    .n = 10,
};

double expect_mean = 5.584602;
double expect_stddev = 3.547868;

#define EQ(X,Y,N) (round((X) * pow(10, N)) == round((Y) * pow(10, N)))

char *test_operations()
{
    int i = 0;
    Stats *st = Stats_create();
    mu_assert(st != NULL, "Failed to create stats.");

    for (i = 0; i < NUM_SAMPLES; i++) {
        Stats_sample(st, samples[i]);
    }

    Stats_dump(st);

    mu_assert(EQ(st->sumsq, expect.sumsq, 3), "sumsq not valid");
    mu_assert(EQ(st->sum, expect.sum, 3), "sum not valid");
    mu_assert(EQ(st->min, expect.min, 3), "min not valid");
    mu_assert(EQ(st->max, expect.max, 3), "max not valid");
    mu_assert(EQ(st->n, expect.n, 3), "max not valid");
    mu_assert(EQ(expect_mean, Stats_mean(st), 3), "mean not valid");
    mu_assert(EQ(expect_stddev, Stats_stddev(st), 3),
            "stddev not valid");

    return NULL;
}

char *test_recreate()
{
    Stats *st = Stats_recreate(
            expect.sum, expect.sumsq, expect.n, expect.min, expect.max);

    mu_assert(st->sum == expect.sum, "sum not equal");
    mu_assert(st->sumsq == expect.sumsq, "sumsq not equal");
    mu_assert(st->n == expect.n, "n not equal");
    mu_assert(st->min == expect.min, "min not equal");
    mu_assert(st->max == expect.max, "max not equal");
    mu_assert(EQ(expect_mean, Stats_mean(st), 3), "mean not valid");
    mu_assert(EQ(expect_stddev, Stats_stddev(st), 3),
            "stddev not valid");

    return NULL;
}

char *all_tests()
{
    mu_suite_start();

    mu_run_test(test_operations);
    mu_run_test(test_recreate);

    return NULL;
}

RUN_TESTS(all_tests);

```

.\ex43\stats.h

```c
#ifndef lcthw_stats_h
#define lcthw_stats_h

typedef struct Stats {
    double sum;
    double sumsq;
    unsigned long n;
    double min;
    double max;
} Stats;

Stats *Stats_recreate(double sum, double sumsq, unsigned long n,
        double min, double max);

Stats *Stats_create();

double Stats_mean(Stats * st);

double Stats_stddev(Stats * st);

void Stats_sample(Stats * st, double s);

void Stats_dump(Stats * st);

#endif

```

.\ex43\stats.c

```c
#include <math.h>
#include <lcthw/stats.h>
#include <stdlib.h>
#include <lcthw/dbg.h>

Stats *Stats_recreate(double sum, double sumsq, unsigned long n,
        double min, double max)
{
    Stats *st = malloc(sizeof(Stats));
    check_mem(st);

    st->sum = sum;
    st->sumsq = sumsq;
    st->n = n;
    st->min = min;
    st->max = max;

    return st;

error:
    return NULL;
}

Stats *Stats_create()
{
    return Stats_recreate(0.0, 0.0, 0L, 0.0, 0.0);
}

double Stats_mean(Stats * st)
{
    return st->sum / st->n;
}

double Stats_stddev(Stats * st)
{
    return sqrt((st->sumsq - (st->sum * st->sum / st->n)) /
            (st->n - 1));
}

void Stats_sample(Stats * st, double s)
{
    st->sum += s;
    st->sumsq += s * s;

    if (st->n == 0) {
        st->min = s;
        st->max = s;
    } else {
        if (st->min > s)
            st->min = s;
        if (st->max < s)
            st->max = s;
    }

    st->n += 1;
}

void Stats_dump(Stats * st)
{
    fprintf(stderr,
            "sum: %f, sumsq: %f, n: %ld, "
            "min: %f, max: %f, mean: %f, stddev: %f",
            st->sum, st->sumsq, st->n, st->min, st->max, Stats_mean(st),
            Stats_stddev(st));
}

```

.\ex43\stats_tests.c

```c
#include "minunit.h"
#include <lcthw/stats.h>
#include <math.h>

const int NUM_SAMPLES = 10;
double samples[] = {
    6.1061334, 9.6783204, 1.2747090, 8.2395131, 0.3333483,
    6.9755066, 1.0626275, 7.6587523, 4.9382973, 9.5788115
};

Stats expect = {
    .sumsq = 425.1641,
    .sum = 55.84602,
    .min = 0.333,
    .max = 9.678,
    .n = 10,
};

double expect_mean = 5.584602;
double expect_stddev = 3.547868;

#define EQ(X,Y,N) (round((X) * pow(10, N)) == round((Y) * pow(10, N)))

char *test_operations()
{
    int i = 0;
    Stats *st = Stats_create();
    mu_assert(st != NULL, "Failed to create stats.");

    for (i = 0; i < NUM_SAMPLES; i++) {
        Stats_sample(st, samples[i]);
    }

    Stats_dump(st);

    mu_assert(EQ(st->sumsq, expect.sumsq, 3), "sumsq not valid");
    mu_assert(EQ(st->sum, expect.sum, 3), "sum not valid");
    mu_assert(EQ(st->min, expect.min, 3), "min not valid");
    mu_assert(EQ(st->max, expect.max, 3), "max not valid");
    mu_assert(EQ(st->n, expect.n, 3), "max not valid");
    mu_assert(EQ(expect_mean, Stats_mean(st), 3), "mean not valid");
    mu_assert(EQ(expect_stddev, Stats_stddev(st), 3),
            "stddev not valid");

    return NULL;
}

char *test_recreate()
{
    Stats *st = Stats_recreate(
            expect.sum, expect.sumsq, expect.n, expect.min, expect.max);

    mu_assert(st->sum == expect.sum, "sum not equal");
    mu_assert(st->sumsq == expect.sumsq, "sumsq not equal");
    mu_assert(st->n == expect.n, "n not equal");
    mu_assert(st->min == expect.min, "min not equal");
    mu_assert(st->max == expect.max, "max not equal");
    mu_assert(EQ(expect_mean, Stats_mean(st), 3), "mean not valid");
    mu_assert(EQ(expect_stddev, Stats_stddev(st), 3),
            "stddev not valid");

    return NULL;
}

char *all_tests()
{
    mu_suite_start();

    mu_run_test(test_operations);
    mu_run_test(test_recreate);

    return NULL;
}

RUN_TESTS(all_tests);

```

The Plan

* A fun and handy little statistics engine for simple analysis.
* Comparing it to the same in R.

Comparing Test vs. R

I'll use R to show you how this works vs. normal calculations using all data.

Breaking It

Easiest way to break this is to just feed it bad data once then the whole
stream is broken.

Extra Credit

* Convert the ``Stats_stddev`` and ``Stats_mean`` to ``static inline`` functions in the ``stats.h`` file instead of in the ``stats.c`` file.
* Use this code to write a performance test of the ``string_algos_test.c``.
  Make it optional, and have it run the base test as a series of samples, and then report
  the results.
* Write a version of this in another programming language you know.  Confirm that this
  version is correct based on what I have here.

Extra Credit

* Write a little program that can take a file full of numbers and spit these statistics
  out for them.
* Make the program accept a table of data that has headers on one line, then all
  of the other numbers on lines after it are separated by any number of spaces.  Your program
  should then print out these statistics for each column by the header name.

### Exercise 44 Ring Buffer

The Plan

Learn about a handy data structure for I/O processing:

Ring Buffers

The Code

.\ex44\netclient.c

```c
#undef NDEBUG
#include <stdlib.h>
#include <sys/select.h>
#include <stdio.h>
#include <lcthw/ringbuffer.h>
#include <lcthw/dbg.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <sys/uio.h>
#include <arpa/inet.h>
#include <netdb.h>
#include <unistd.h>
#include <fcntl.h>

struct tagbstring NL = bsStatic("\n");
struct tagbstring CRLF = bsStatic("\r\n");

int nonblock(int fd)
{
    int flags = fcntl(fd, F_GETFL, 0);
    check(flags >= 0, "Invalid flags on nonblock.");

    int rc = fcntl(fd, F_SETFL, flags | O_NONBLOCK);
    check(rc == 0, "Can't set nonblocking.");

    return 0;
error:
    return -1;
}

int client_connect(char *host, char *port)
{
    int rc = 0;
    struct addrinfo *addr = NULL;

    rc = getaddrinfo(host, port, NULL, &addr);
    check(rc == 0, "Failed to lookup %s:%s", host, port);

    int sock = socket(AF_INET, SOCK_STREAM, 0);
    check(sock >= 0, "Cannot create a socket.");

    rc = connect(sock, addr->ai_addr, addr->ai_addrlen);
    check(rc == 0, "Connect failed.");

    rc = nonblock(sock);
    check(rc == 0, "Can't set nonblocking.");

    freeaddrinfo(addr);
    return sock;

error:
    freeaddrinfo(addr);
    return -1;
}

int read_some(RingBuffer * buffer, int fd, int is_socket)
{
    int rc = 0;

    if (RingBuffer_available_data(buffer) == 0) {
        buffer->start = buffer->end = 0;
    }

    if (is_socket) {
        rc = recv(fd, RingBuffer_starts_at(buffer),
                RingBuffer_available_space(buffer), 0);
    } else {
        rc = read(fd, RingBuffer_starts_at(buffer),
                RingBuffer_available_space(buffer));
    }

    check(rc >= 0, "Failed to read from fd: %d", fd);

    RingBuffer_commit_write(buffer, rc);

    return rc;

error:
    return -1;
}

int write_some(RingBuffer * buffer, int fd, int is_socket)
{
    int rc = 0;
    bstring data = RingBuffer_get_all(buffer);

    check(data != NULL, "Failed to get from the buffer.");
    check(bfindreplace(data, &NL, &CRLF, 0) == BSTR_OK,
            "Failed to replace NL.");

    if (is_socket) {
        rc = send(fd, bdata(data), blength(data), 0);
    } else {
        rc = write(fd, bdata(data), blength(data));
    }

    check(rc == blength(data), "Failed to write everything to fd: %d.",
            fd);
    bdestroy(data);

    return rc;

error:
    return -1;
}

int main(int argc, char *argv[])
{
    fd_set allreads;
    fd_set readmask;

    int socket = 0;
    int rc = 0;
    RingBuffer *in_rb = RingBuffer_create(1024 * 10);
    RingBuffer *sock_rb = RingBuffer_create(1024 * 10);

    check(argc == 3, "USAGE: netclient host port");

    socket = client_connect(argv[1], argv[2]);
    check(socket >= 0, "connect to %s:%s failed.", argv[1], argv[2]);

    FD_ZERO(&allreads);
    FD_SET(socket, &allreads);
    FD_SET(0, &allreads);

    while (1) {
        readmask = allreads;
        rc = select(socket + 1, &readmask, NULL, NULL, NULL);
        check(rc >= 0, "select failed.");

        if (FD_ISSET(0, &readmask)) {
            rc = read_some(in_rb, 0, 0);
            check_debug(rc != -1, "Failed to read from stdin.");
        }

        if (FD_ISSET(socket, &readmask)) {
            rc = read_some(sock_rb, socket, 0);
            check_debug(rc != -1, "Failed to read from socket.");
        }

        while (!RingBuffer_empty(sock_rb)) {
            rc = write_some(sock_rb, 1, 0);
            check_debug(rc != -1, "Failed to write to stdout.");
        }

        while (!RingBuffer_empty(in_rb)) {
            rc = write_some(in_rb, socket, 1);
            check_debug(rc != -1, "Failed to write to socket.");
        }
    }

    return 0;

error:
    return -1;
}

```

It's basically a DArray with dynamic start and end settings.
You can *also* use a Queue of bstrings to do almost the same thing.

Code Review

.\ex44\netclient.c

```c
#undef NDEBUG
#include <stdlib.h>
#include <sys/select.h>
#include <stdio.h>
#include <lcthw/ringbuffer.h>
#include <lcthw/dbg.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <sys/uio.h>
#include <arpa/inet.h>
#include <netdb.h>
#include <unistd.h>
#include <fcntl.h>

struct tagbstring NL = bsStatic("\n");
struct tagbstring CRLF = bsStatic("\r\n");

int nonblock(int fd)
{
    int flags = fcntl(fd, F_GETFL, 0);
    check(flags >= 0, "Invalid flags on nonblock.");

    int rc = fcntl(fd, F_SETFL, flags | O_NONBLOCK);
    check(rc == 0, "Can't set nonblocking.");

    return 0;
error:
    return -1;
}

int client_connect(char *host, char *port)
{
    int rc = 0;
    struct addrinfo *addr = NULL;

    rc = getaddrinfo(host, port, NULL, &addr);
    check(rc == 0, "Failed to lookup %s:%s", host, port);

    int sock = socket(AF_INET, SOCK_STREAM, 0);
    check(sock >= 0, "Cannot create a socket.");

    rc = connect(sock, addr->ai_addr, addr->ai_addrlen);
    check(rc == 0, "Connect failed.");

    rc = nonblock(sock);
    check(rc == 0, "Can't set nonblocking.");

    freeaddrinfo(addr);
    return sock;

error:
    freeaddrinfo(addr);
    return -1;
}

int read_some(RingBuffer * buffer, int fd, int is_socket)
{
    int rc = 0;

    if (RingBuffer_available_data(buffer) == 0) {
        buffer->start = buffer->end = 0;
    }

    if (is_socket) {
        rc = recv(fd, RingBuffer_starts_at(buffer),
                RingBuffer_available_space(buffer), 0);
    } else {
        rc = read(fd, RingBuffer_starts_at(buffer),
                RingBuffer_available_space(buffer));
    }

    check(rc >= 0, "Failed to read from fd: %d", fd);

    RingBuffer_commit_write(buffer, rc);

    return rc;

error:
    return -1;
}

int write_some(RingBuffer * buffer, int fd, int is_socket)
{
    int rc = 0;
    bstring data = RingBuffer_get_all(buffer);

    check(data != NULL, "Failed to get from the buffer.");
    check(bfindreplace(data, &NL, &CRLF, 0) == BSTR_OK,
            "Failed to replace NL.");

    if (is_socket) {
        rc = send(fd, bdata(data), blength(data), 0);
    } else {
        rc = write(fd, bdata(data), blength(data));
    }

    check(rc == blength(data), "Failed to write everything to fd: %d.",
            fd);
    bdestroy(data);

    return rc;

error:
    return -1;
}

int main(int argc, char *argv[])
{
    fd_set allreads;
    fd_set readmask;

    int socket = 0;
    int rc = 0;
    RingBuffer *in_rb = RingBuffer_create(1024 * 10);
    RingBuffer *sock_rb = RingBuffer_create(1024 * 10);

    check(argc == 3, "USAGE: netclient host port");

    socket = client_connect(argv[1], argv[2]);
    check(socket >= 0, "connect to %s:%s failed.", argv[1], argv[2]);

    FD_ZERO(&allreads);
    FD_SET(socket, &allreads);
    FD_SET(0, &allreads);

    while (1) {
        readmask = allreads;
        rc = select(socket + 1, &readmask, NULL, NULL, NULL);
        check(rc >= 0, "select failed.");

        if (FD_ISSET(0, &readmask)) {
            rc = read_some(in_rb, 0, 0);
            check_debug(rc != -1, "Failed to read from stdin.");
        }

        if (FD_ISSET(socket, &readmask)) {
            rc = read_some(sock_rb, socket, 0);
            check_debug(rc != -1, "Failed to read from socket.");
        }

        while (!RingBuffer_empty(sock_rb)) {
            rc = write_some(sock_rb, 1, 0);
            check_debug(rc != -1, "Failed to write to stdout.");
        }

        while (!RingBuffer_empty(in_rb)) {
            rc = write_some(in_rb, socket, 1);
            check_debug(rc != -1, "Failed to write to socket.");
        }
    }

    return 0;

error:
    return -1;
}

```

The Analysis

* Watch a ring buffer work in the debugger.
* Draw it visually to explore it.
* The purpose is to efficiently add and remove data when the amount added and removed is random.

Pause!

I will next review the unit test I wrote so if you want to attempt
solving it yourself then pause now.

The Unit Test

Here's my version of the unit test.

Breaking It

* The biggest mistake you'll make with a ring buffer is off-by-one errors.
* This is why the RingBuffer\_commit\_ and other macros exist.
* Another common mistake is to use it between threads, but that's a whole other book.

Extra Credit

* Create an alternative implementation of ``RingBuffer`` that uses
  the POSIX trick and a unit test for it.
* Add a performance comparison test to this unit test that compares the
  two versions by fuzzing them with random data and random read/write operations.
  Make sure that you set up this fuzzing so that the same operations are done
  to each version, and you can compare them between runs.

### Exercise 45 A Simple TCP/IP Client

The Plan

* Learn to use the *select* method and a RingBuffer to write a simple command line network client.

How select Works

Code Review

Improving It

These read functions are useful so I can put them in RingBuffer.

Extra Credit

* As I mentioned, there are quite a few functions you may not know, so
  look them up.  In fact, look them all up even if you think you know
  them.
* Go back through and add various defensive programming checks to
  the functions to improve them.
* Use the *getopt* function to allow the user
  the option *not* to translate *\n* to *\r\n*. This
  is only needed on protocols that require it for line endings, like HTTP.
  Sometimes you don't want the translation, so give the user the option.

### Exercise 46 Ternary Search Tree

The Plan

Learn about my favorite data structure ever:

Ternary Search Tree

The Code

.\ex46\tstree.h

```c
#ifndef _lcthw_TSTree_h
#define _lcthw_TSTree_h

#include <stdlib.h>
#include <lcthw/darray.h>

typedef struct TSTree {
    char splitchar;
    struct TSTree *low;
    struct TSTree *equal;
    struct TSTree *high;
    void *value;
} TSTree;

void *TSTree_search(TSTree * root, const char *key, size_t len);

void *TSTree_search_prefix(TSTree * root, const char *key, size_t len);

typedef void (*TSTree_traverse_cb) (void *value, void *data);

TSTree *TSTree_insert(TSTree * node, const char *key, size_t len,
        void *value);

void TSTree_traverse(TSTree * node, TSTree_traverse_cb cb, void *data);

void TSTree_destroy(TSTree * root);

#endif

```

.\ex46\tstree.c

```c
#include <stdlib.h>
#include <stdio.h>
#include <assert.h>
#include <lcthw/dbg.h>
#include <lcthw/tstree.h>

static inline TSTree *TSTree_insert_base(TSTree * root, TSTree * node,
        const char *key, size_t len,
        void *value)
{
    if (node == NULL) {
        node = (TSTree *) calloc(1, sizeof(TSTree));

        if (root == NULL) {
            root = node;
        }

        node->splitchar = *key;
    }

    if (*key < node->splitchar) {
        node->low = TSTree_insert_base(
                root, node->low, key, len, value);
    } else if (*key == node->splitchar) {
        if (len > 1) {
            node->equal = TSTree_insert_base(
                    root, node->equal, key + 1, len - 1, value);
        } else {
            assert(node->value == NULL && "Duplicate insert into tst.");
            node->value = value;
        }
    } else {
        node->high = TSTree_insert_base(
                root, node->high, key, len, value);
    }

    return node;
}

TSTree *TSTree_insert(TSTree * node, const char *key, size_t len,
        void *value)
{
    return TSTree_insert_base(node, node, key, len, value);
}

void *TSTree_search(TSTree * root, const char *key, size_t len)
{
    TSTree *node = root;
    size_t i = 0;

    while (i < len && node) {
        if (key[i] < node->splitchar) {
            node = node->low;
        } else if (key[i] == node->splitchar) {
            i++;
            if (i < len)
                node = node->equal;
        } else {
            node = node->high;
        }
    }

    if (node) {
        return node->value;
    } else {
        return NULL;
    }
}

void *TSTree_search_prefix(TSTree * root, const char *key, size_t len)
{
    if (len == 0)
        return NULL;

    TSTree *node = root;
    TSTree *last = NULL;
    size_t i = 0;

    while (i < len && node) {
        if (key[i] < node->splitchar) {
            node = node->low;
        } else if (key[i] == node->splitchar) {
            i++;
            if (i < len) {
                if (node->value)
                    last = node;
                node = node->equal;
            }
        } else {
            node = node->high;
        }
    }

    node = node ? node : last;

    // traverse until we find the first value in the equal chain
    // this is then the first node with this prefix
    while (node && !node->value) {
        node = node->equal;
    }

    return node ? node->value : NULL;
}

void TSTree_traverse(TSTree * node, TSTree_traverse_cb cb, void *data)
{
    if (!node)
        return;

    if (node->low)
        TSTree_traverse(node->low, cb, data);

    if (node->equal) {
        TSTree_traverse(node->equal, cb, data);
    }

    if (node->high)
        TSTree_traverse(node->high, cb, data);

    if (node->value)
        cb(node->value, data);
}

void TSTree_destroy(TSTree * node)
{
    if (node == NULL)
        return;

    if (node->low)
        TSTree_destroy(node->low);

    if (node->equal) {
        TSTree_destroy(node->equal);
    }

    if (node->high)
        TSTree_destroy(node->high);

    free(node);
}

```

.\ex46\tstree_tests.c

```c
#include "minunit.h"
#include <lcthw/tstree.h>
#include <string.h>
#include <assert.h>
#include <lcthw/bstrlib.h>

TSTree *node = NULL;
char *valueA = "VALUEA";
char *valueB = "VALUEB";
char *value2 = "VALUE2";
char *value4 = "VALUE4";
char *reverse = "VALUER";
int traverse_count = 0;

struct tagbstring test1 = bsStatic("TEST");
struct tagbstring test2 = bsStatic("TEST2");
struct tagbstring test3 = bsStatic("TSET");
struct tagbstring test4 = bsStatic("T");

char *test_insert()
{
    node = TSTree_insert(node, bdata(&test1), blength(&test1), valueA);
    mu_assert(node != NULL, "Failed to insert into tst.");

    node = TSTree_insert(node, bdata(&test2), blength(&test2), value2);
    mu_assert(node != NULL,
            "Failed to insert into tst with second name.");

    node = TSTree_insert(node, bdata(&test3), blength(&test3), reverse);
    mu_assert(node != NULL,
            "Failed to insert into tst with reverse name.");

    node = TSTree_insert(node, bdata(&test4), blength(&test4), value4);
    mu_assert(node != NULL,
            "Failed to insert into tst with second name.");

    return NULL;
}

char *test_search_exact()
{
    // tst returns the last one inserted
    void *res = TSTree_search(node, bdata(&test1), blength(&test1));
    mu_assert(res == valueA,
            "Got the wrong value back, should get A not B.");

    // tst does not find if not exact
    res = TSTree_search(node, "TESTNO", strlen("TESTNO"));
    mu_assert(res == NULL, "Should not find anything.");

    return NULL;
}

char *test_search_prefix()
{
    void *res = TSTree_search_prefix(
            node, bdata(&test1), blength(&test1));
    debug("result: %p, expected: %p", res, valueA);
    mu_assert(res == valueA, "Got wrong valueA by prefix.");

    res = TSTree_search_prefix(node, bdata(&test1), 1);
    debug("result: %p, expected: %p", res, valueA);
    mu_assert(res == value4, "Got wrong value4 for prefix of 1.");

    res = TSTree_search_prefix(node, "TE", strlen("TE"));
    mu_assert(res != NULL, "Should find for short prefix.");

    res = TSTree_search_prefix(node, "TE--", strlen("TE--"));
    mu_assert(res != NULL, "Should find for partial prefix.");

    return NULL;
}

void TSTree_traverse_test_cb(void *value, void *data)
{
    assert(value != NULL && "Should not get NULL value.");
    assert(data == valueA && "Expecting valueA as the data.");
    traverse_count++;
}

char *test_traverse()
{
    traverse_count = 0;
    TSTree_traverse(node, TSTree_traverse_test_cb, valueA);
    debug("traverse count is: %d", traverse_count);
    mu_assert(traverse_count == 4, "Didn't find 4 keys.");

    return NULL;
}

char *test_destroy()
{
    TSTree_destroy(node);

    return NULL;
}

char *all_tests()
{
    mu_suite_start();

    mu_run_test(test_insert);
    mu_run_test(test_search_exact);
    mu_run_test(test_search_prefix);
    mu_run_test(test_traverse);
    mu_run_test(test_destroy);

    return NULL;
}

RUN_TESTS(all_tests);

```

Similar to a Binary Search Tree, but it has 3 branches per node based on
the characters in strings.

Advantages

* Find any string comparing at most N characters.
* Detect *missing* strings as fast, usually faster.
* Find all strings that start with, or contain, any substring as fast.
* Find all similar known strings quickly.

Disadvantages

* Delete is a pain, as in most trees.
* Uses lots of memory to store keys, so bad for sets of large keys.
* Kind of weird for most programmers.

Improving It

* You could allow duplicates by using a *DArray* instead of the
  *value*.
* As I mentioned earlier, deleting is hard, but you could simulate it by setting
  the values to *NULL* so that they are effectively gone.
* There are no ways to collect all of the possible matching values.  I'll have
  you implement that in an extra credit.
* There are other algorithms that are more complex but have slightly
  better properties.  Take a look at suffix array, suffix tree, and
  radix tree structures.

Extra Credit

* Implement a *TSTree_collect* that returns a *DArray* containing
  all of the keys that match the given prefix.
* Implement *TSTree_search_suffix* and a *TSTree_insert_suffix*
  so you can do suffix searches and inserts.
* Use the debugger to see how this structure is used in memory 
  compared to the *BSTree* and *Hashmap*.

### Exercise 47 A Fast URL Router

The Plan

Use the *TSTree* to do something useful:

Route URLs

.\ex47\ex47_urls.txt

```
/test.tst TestHandler
/ IndexHandler
/test/this/out/index.html PageHandler
/index.html PageHandler
/and/then/i/have/things/to/test.html PageHandler

```

Code Review

.\ex47\urlor.c

```c
#include <lcthw/tstree.h>
#include <lcthw/bstrlib.h>

TSTree *add_route_data(TSTree * routes, bstring line)
{
    struct bstrList *data = bsplit(line, ' ');
    check(data->qty == 2, "Line '%s' does not have 2 columns",
            bdata(line));

    routes = TSTree_insert(routes,
            bdata(data->entry[0]),
            blength(data->entry[0]),
            bstrcpy(data->entry[1]));

    bstrListDestroy(data);

    return routes;

error:
    return NULL;
}

TSTree *load_routes(const char *file)
{
    TSTree *routes = NULL;
    bstring line = NULL;
    FILE *routes_map = NULL;

    routes_map = fopen(file, "r");
    check(routes_map != NULL, "Failed to open routes: %s", file);

    while ((line = bgets((bNgetc) fgetc, routes_map, '\n')) != NULL) {
        check(btrimws(line) == BSTR_OK, "Failed to trim line.");
        routes = add_route_data(routes, line);
        check(routes != NULL, "Failed to add route.");
        bdestroy(line);
    }

    fclose(routes_map);
    return routes;

error:
    if (routes_map) fclose(routes_map);
    if (line) bdestroy(line);

    return NULL;
}

bstring match_url(TSTree * routes, bstring url)
{
    bstring route = TSTree_search(routes, bdata(url), blength(url));

    if (route == NULL) {
        printf("No exact match found, trying prefix.\n");
        route = TSTree_search_prefix(routes, bdata(url), blength(url));
    }

    return route;
}

bstring read_line(const char *prompt)
{
    printf("%s", prompt);

    bstring result = bgets((bNgetc) fgetc, stdin, '\n');
    check_debug(result != NULL, "stdin closed.");

    check(btrimws(result) == BSTR_OK, "Failed to trim.");

    return result;

error:
    return NULL;
}

void bdestroy_cb(void *value, void *ignored)
{
    (void)ignored;
    bdestroy((bstring) value);
}

void destroy_routes(TSTree * routes)
{
    TSTree_traverse(routes, bdestroy_cb, NULL);
    TSTree_destroy(routes);
}

int main(int argc, char *argv[])
{
    bstring url = NULL;
    bstring route = NULL;
    TSTree *routes = NULL;

    check(argc == 2, "USAGE: urlor <urlfile>");

    routes = load_routes(argv[1]);
    check(routes != NULL, "Your route file has an error.");

    while (1) {
        url = read_line("URL> ");
        check_debug(url != NULL, "goodbye.");

        route = match_url(routes, url);

        if (route) {
            printf("MATCH: %s == %s\n", bdata(url), bdata(route));
        } else {
            printf("FAIL: %s\n", bdata(url));
        }

        bdestroy(url);
    }

    destroy_routes(routes);
    return 0;

error:
    destroy_routes(routes);
    return 1;
}

```

The Analysis

Watch me play with it and then tell you how it's working.

Improving It

* Collect all possible matches then choose the longest as winner.
* Use TSTree to find prefixes, then regex to choose winner.

Extra Credit

* Instead of just storing the string for the handler, create an actual engine that uses a
  *Handler* struct to store the application.  The structure would store the URL to which it's attached, the name, and anything else you'd need to make an actual routing system.

Extra Credit

* Instead of mapping URLs to arbitrary names, map them to .so files and use the *dlopen*
  system to load handlers on the fly and call callbacks they contain.  Put these callbacks that
  in your *Handler* struct, and then you have yourself a fully dynamic callback
  handler system in C.

### Exercise 48a A Simple Network Server:

Project Description

The Plan

Start your first long running project:

statserve

The Purpose

You'll get the project started and get a minimum first hack going.

The Requirements

1. Create a simple network server that accepts a connection on port 7899 from
   *netclient* or the *nc* command, and echoes back anything you type.
2. You'll need to learn how to bind a port, listen on the socket, and answer it.
   Use your research skills to study how this is done and attempt to implement it
   yourself.

The Requirements

3. The more important part of this project is laying out the project directory
   from the *c-skeleton*, and making sure you can build everything and get it
   working.
4. Don't worry about things like daemons or anything else.  Your server just has
   to run from the command line and keep running.

The Clues

I will now give you some clues:

* USE liblcthw!
* Remember you did a client already, you just need to make a server.
* Do NOT use select! Use fork() for the server.
* Keep it *simple*.  Don't worry about anything other than accepting a connection and closing.
* Stay small, build slowly.

Important References

* Research online for "echo server in C".
* Read man (2) pages for *accept*, *bind*, *listen*, *connect*, *select*, *socket*, and *shutdown*.

Encouragement

This will be *HARD*!  Try it your best, and take it piece by piece.  You can do it, but remember if you give up the next video (48b) will show you the code to my solution and how to solve it.  You can peek there then come back when you're stuck.

### Exercise 48b A Simple Network Server:
.\ex48b\c-skeleton

.\ex48b\c-skeleton\src\dbg.h

```c
#ifndef __dbg_h__
#define __dbg_h__

#include <stdio.h>
#include <errno.h>
#include <string.h>

#ifdef NDEBUG
#define debug(M, ...)
#else
#define debug(M, ...) fprintf(stderr, "DEBUG %s:%d: " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)
#endif

#define clean_errno() (errno == 0 ? "None" : strerror(errno))

#define log_err(M, ...) fprintf(stderr,\
        "[ERROR] (%s:%d: errno: %s) " M "\n", __FILE__, __LINE__,\
        clean_errno(), ##__VA_ARGS__)

#define log_warn(M, ...) fprintf(stderr,\
        "[WARN] (%s:%d: errno: %s) " M "\n",\
        __FILE__, __LINE__, clean_errno(), ##__VA_ARGS__)

#define log_info(M, ...) fprintf(stderr, "[INFO] (%s:%d) " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)

#define check(A, M, ...) if(!(A)) {\
    log_err(M, ##__VA_ARGS__); errno=0; goto error; }

#define sentinel(M, ...)  { log_err(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#define check_mem(A) check((A), "Out of memory.")

#define check_debug(A, M, ...) if(!(A)) { debug(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#endif

```

.\ex48b\c-skeleton\src\libex29.c

```c
#include <stdio.h>
#include <ctype.h>
#include "dbg.h"

int print_a_message(const char *msg)
{
    printf("A STRING: %s\n", msg);

    return 0;
}

int uppercase(const char *msg)
{
    int i = 0;

    // BUG: \0 termination problems
    for(i = 0; msg[i] != '\0'; i++) {
        printf("%c", toupper(msg[i]));
    }

    printf("\n");

    return 0;
}

int lowercase(const char *msg)
{
    int i = 0;

    // BUG: \0 termination problems
    for(i = 0; msg[i] != '\0'; i++) {
        printf("%c", tolower(msg[i]));
    }

    printf("\n");

    return 0;
}

int fail_on_purpose(const char *msg)
{
    return 1;
} 

```

.\ex48b\c-skeleton\tests\libex29_tests.c

```c
#include "minunit.h"
#include <dlfcn.h>

typedef int (*lib_function) (const char *data);
char *lib_file = "build/libYOUR_LIBRARY.so";
void *lib = NULL;

int check_function(const char *func_to_run, const char *data,
        int expected)
{
    lib_function func = dlsym(lib, func_to_run);
    check(func != NULL,
            "Did not find %s function in the library %s: %s", func_to_run,
            lib_file, dlerror());

    int rc = func(data);
    check(rc == expected, "Function %s return %d for data: %s",
            func_to_run, rc, data);

    return 1;
error:
    return 0;
}

char *test_dlopen()
{
    lib = dlopen(lib_file, RTLD_NOW);
    mu_assert(lib != NULL, "Failed to open the library to test.");

    return NULL;
}

char *test_functions()
{
    mu_assert(check_function("print_a_message", "Hello", 0),
            "print_a_message failed.");
    mu_assert(check_function("uppercase", "Hello", 0),
            "uppercase failed.");
    mu_assert(check_function("lowercase", "Hello", 0),
            "lowercase failed.");

    return NULL;
}

char *test_failures()
{
    mu_assert(check_function("fail_on_purpose", "Hello", 1),
            "fail_on_purpose should fail.");

    return NULL;
}

char *test_dlclose()
{
    int rc = dlclose(lib);
    mu_assert(rc == 0, "Failed to close lib.");

    return NULL;
}

char *all_tests()
{
    mu_suite_start();

    mu_run_test(test_dlopen);
    mu_run_test(test_functions);
    mu_run_test(test_failures);
    mu_run_test(test_dlclose);

    return NULL;
}

RUN_TESTS(all_tests);

```

.\ex48b\statserve

.\ex48b\statserve\bin\statserve.c

```c
#include <stdio.h>
#include <lcthw/dbg.h>
#include "statserve.h"
#include "net.h"

int main(int argc, char *argv[])
{
    check(argc == 3, "USAGE: statserve host port");

    const char *host = argv[1];
    const char *port = argv[2];

    check(echo_server(host, port), "Failed to run the echo server.");

    return 0;

error:

    return 1;
}

```

.\ex48b\statserve\src\dbg.h

```c
#ifndef __dbg_h__
#define __dbg_h__

#include <stdio.h>
#include <errno.h>
#include <string.h>

#ifdef NDEBUG
#define debug(M, ...)
#else
#define debug(M, ...) fprintf(stderr, "DEBUG %s:%d: " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)
#endif

#define clean_errno() (errno == 0 ? "None" : strerror(errno))

#define log_err(M, ...) fprintf(stderr,\
        "[ERROR] (%s:%d: errno: %s) " M "\n", __FILE__, __LINE__,\
        clean_errno(), ##__VA_ARGS__)

#define log_warn(M, ...) fprintf(stderr,\
        "[WARN] (%s:%d: errno: %s) " M "\n",\
        __FILE__, __LINE__, clean_errno(), ##__VA_ARGS__)

#define log_info(M, ...) fprintf(stderr, "[INFO] (%s:%d) " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)

#define check(A, M, ...) if(!(A)) {\
    log_err(M, ##__VA_ARGS__); errno=0; goto error; }

#define sentinel(M, ...)  { log_err(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#define check_mem(A) check((A), "Out of memory.")

#define check_debug(A, M, ...) if(!(A)) { debug(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#endif

```

.\ex48b\statserve\src\net.c

```c
#include <stdlib.h>
#include <sys/select.h>
#include <stdio.h>
#include <lcthw/ringbuffer.h>
#include <lcthw/dbg.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <sys/uio.h>
#include <arpa/inet.h>
#include <netdb.h>
#include <unistd.h>
#include <fcntl.h>
#include "net.h"

struct tagbstring NL = bsStatic("\n");
struct tagbstring CRLF = bsStatic("\r\n");

int nonblock(int fd)
{
    int flags = fcntl(fd, F_GETFL, 0);
    check(flags >= 0, "Invalid flags on nonblock.");

    int rc = fcntl(fd, F_SETFL, flags | O_NONBLOCK);
    check(rc == 0, "Can't set nonblocking.");

    return 0;
error:
    return -1;
}

int client_connect(char *host, char *port)
{
    int rc = 0;
    struct addrinfo *addr = NULL;

    rc = getaddrinfo(host, port, NULL, &addr);
    check(rc == 0, "Failed to lookup %s:%s", host, port);

    int sock = socket(AF_INET, SOCK_STREAM, 0);
    check(sock >= 0, "Cannot create a socket.");

    rc = connect(sock, addr->ai_addr, addr->ai_addrlen);
    check(rc == 0, "Connect failed.");

    rc = nonblock(sock);
    check(rc == 0, "Can't set nonblocking.");

    freeaddrinfo(addr);
    return sock;

error:
    freeaddrinfo(addr);
    return -1;
}

int read_some(RingBuffer * buffer, int fd, int is_socket)
{
    int rc = 0;

    if (RingBuffer_available_data(buffer) == 0) {
        buffer->start = buffer->end = 0;
    }

    if (is_socket) {
        rc = recv(fd, RingBuffer_starts_at(buffer),
                RingBuffer_available_space(buffer), 0);
    } else {
        rc = read(fd, RingBuffer_starts_at(buffer),
                RingBuffer_available_space(buffer));
    }

    check(rc >= 0, "Failed to read from fd: %d", fd);

    RingBuffer_commit_write(buffer, rc);

    return rc;

error:
    return -1;
}

int write_some(RingBuffer * buffer, int fd, int is_socket)
{
    int rc = 0;
    bstring data = RingBuffer_get_all(buffer);

    check(data != NULL, "Failed to get from the buffer.");
    check(bfindreplace(data, &NL, &CRLF, 0) == BSTR_OK,
            "Failed to replace NL.");

    if (is_socket) {
        rc = send(fd, bdata(data), blength(data), 0);
    } else {
        rc = write(fd, bdata(data), blength(data));
    }

    check(rc == blength(data), "Failed to write everything to fd: %d.",
            fd);
    bdestroy(data);

    return rc;

error:
    return -1;
}

int attempt_listen(struct addrinfo *info)
{
    int sockfd = -1; // default fail
    int rc = -1;
    int yes = 1;

    check(info != NULL, "Invalid addrinfo.");

    // create a socket with the addrinfo
    sockfd = socket(info->ai_family, info->ai_socktype,
            info->ai_protocol);
    check_debug(sockfd != -1, "Failed to bind to address. Trying more.");

    // set the SO_REUSEADDR option on the socket
    rc = setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(int));
    check_debug(rc == 0, "Failed to set SO_REUSADDR.");

    // attempt to bind to it
    rc = bind(sockfd, info->ai_addr, info->ai_addrlen);
    check_debug(rc == 0, "Failed to find socket.");

    // finally listen with a backlog
    rc = listen(sockfd, BACKLOG);
    check_debug(rc == 0, "Failed to listen to socket.");

    return sockfd;

error:
    return -1;
}

int server_listen(const char *host, const char *port)
{
    int rc = 0;
    int sockfd = -1; // default fail value
    struct addrinfo *info = NULL;
    struct addrinfo *next_p = NULL;
    struct addrinfo addr = {
        .ai_family = AF_UNSPEC,
        .ai_socktype = SOCK_STREAM,
        .ai_flags = AI_PASSIVE
    };

    check(host != NULL, "Invalid host.");
    check(port != NULL, "Invalid port.");

    // get the address info for host and port
    rc = getaddrinfo(NULL, port, &addr, &info);
    check(rc == 0, "Failed to get address info for connect.");

    // cycle through the available list to find one
    for(next_p = info; next_p != NULL; next_p = next_p->ai_next)
    {
        // attempt to listen to each one
        sockfd = attempt_listen(next_p);
        if(sockfd != -1) break;
    }

    // either we found one and were able to listen or nothing.
    check(sockfd != -1, "All possible addresses failed.");

error: //fallthrough
    if(info) freeaddrinfo(info);
    // this gets set by the above to either -1 or valid
    return sockfd;
}

```

.\ex48b\statserve\src\net.h

```c
#ifndef _net_h
#define _net_h

#include <lcthw/ringbuffer.h>

#define BACKLOG 10

int nonblock(int fd);
int client_connect(char *host, char *port);
int read_some(RingBuffer * buffer, int fd, int is_socket);
int write_some(RingBuffer * buffer, int fd, int is_socket);
int server_listen(const char *host, const char *port);

#endif

```

.\ex48b\statserve\src\statserve.c

```c
#include <stdio.h>
#include <ctype.h>
#include <lcthw/dbg.h>
#include <unistd.h>
#include <stdlib.h>
#include <signal.h>
#include <sys/wait.h>
#include "net.h"
#include <netdb.h>

const int RB_SIZE = 1024 * 10;

void handle_sigchild(int sig) {
    sig = 0; // ignore it
    while(waitpid(-1, NULL, WNOHANG) > 0) {
    }
}

void client_handler(int client_fd)
{
    int rc = 0;
    // need a ringbuffer for the input
    RingBuffer *sock_rb = RingBuffer_create(RB_SIZE);

    // read_some in a loop
    while(read_some(sock_rb, client_fd, 1) != -1) {
        // write_it back off the ringbuffer
        if(write_some(sock_rb, client_fd, 1) == -1) {
            debug("Client closed.");
            break;
        }
    }

    // close the socket
    rc = close(client_fd);
    check(rc != -1, "Failed to close the socket.");

error: // fallthrough
    if(sock_rb) RingBuffer_destroy(sock_rb);
    exit(0); // just exit the child process
}

int echo_server(const char *host, const char *port)
{
    int rc = 0;
    struct sockaddr_in client_addr;
    socklen_t sin_size = sizeof(client_addr);
    int server_socket = 0;
    int client_fd = 0;

    struct sigaction sa = {
        .sa_handler = handle_sigchild,
        .sa_flags = SA_RESTART | SA_NOCLDSTOP
    };

    check(host != NULL, "Invalid host.");
    check(port != NULL, "Invalid port.");

    // create a sigaction that handles SIGCHLD
    sigemptyset(&sa.sa_mask);
    rc = sigaction(SIGCHLD, &sa, 0);
    check(rc != -1, "Failed to setup signal handler for child processes.");

    // listen on the given port and host
    server_socket = server_listen(host, port);
    check(server_socket >= 0, "bind to %s:%s failed.", host, port);

    while(1) {
        // accept the connection
        client_fd = accept(server_socket, (struct sockaddr *)&client_addr, &sin_size); 
        check(client_fd >= 0, "Failed to accept connection.");

        debug("Client connected.");

        rc = fork();
        if(rc == 0) {
            // child process
            close(server_socket); // don't need this
            // handle the client
            client_handler(client_fd);
        } else {
            // server process
            close(client_fd); // don't need this
        }
    }

error:  // fallthrough
    return -1;
}

```

.\ex48b\statserve\src\statserve.h

```c
#ifndef _statserve_h
#define _statserve_h

int echo_server(const char *host, const char *port);

#endif

```

.\ex48b\statserve\tests\statserve_tests.c

```c
#include "minunit.h"
#include <dlfcn.h>
#include "statserve.h"

char *test_dummy()
{
    return NULL;
}

char *all_tests()
{
    mu_suite_start();

    mu_run_test(test_dummy);

    return NULL;
}

RUN_TESTS(all_tests);

```

.\ex48b\c-skeleton

.\ex48b\c-skeleton\src\dbg.h

```c
#ifndef __dbg_h__
#define __dbg_h__

#include <stdio.h>
#include <errno.h>
#include <string.h>

#ifdef NDEBUG
#define debug(M, ...)
#else
#define debug(M, ...) fprintf(stderr, "DEBUG %s:%d: " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)
#endif

#define clean_errno() (errno == 0 ? "None" : strerror(errno))

#define log_err(M, ...) fprintf(stderr,\
        "[ERROR] (%s:%d: errno: %s) " M "\n", __FILE__, __LINE__,\
        clean_errno(), ##__VA_ARGS__)

#define log_warn(M, ...) fprintf(stderr,\
        "[WARN] (%s:%d: errno: %s) " M "\n",\
        __FILE__, __LINE__, clean_errno(), ##__VA_ARGS__)

#define log_info(M, ...) fprintf(stderr, "[INFO] (%s:%d) " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)

#define check(A, M, ...) if(!(A)) {\
    log_err(M, ##__VA_ARGS__); errno=0; goto error; }

#define sentinel(M, ...)  { log_err(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#define check_mem(A) check((A), "Out of memory.")

#define check_debug(A, M, ...) if(!(A)) { debug(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#endif

```

.\ex48b\c-skeleton\src\libex29.c

```c
#include <stdio.h>
#include <ctype.h>
#include "dbg.h"

int print_a_message(const char *msg)
{
    printf("A STRING: %s\n", msg);

    return 0;
}

int uppercase(const char *msg)
{
    int i = 0;

    // BUG: \0 termination problems
    for(i = 0; msg[i] != '\0'; i++) {
        printf("%c", toupper(msg[i]));
    }

    printf("\n");

    return 0;
}

int lowercase(const char *msg)
{
    int i = 0;

    // BUG: \0 termination problems
    for(i = 0; msg[i] != '\0'; i++) {
        printf("%c", tolower(msg[i]));
    }

    printf("\n");

    return 0;
}

int fail_on_purpose(const char *msg)
{
    return 1;
} 

```

.\ex48b\c-skeleton\tests\libex29_tests.c

```c
#include "minunit.h"
#include <dlfcn.h>

typedef int (*lib_function) (const char *data);
char *lib_file = "build/libYOUR_LIBRARY.so";
void *lib = NULL;

int check_function(const char *func_to_run, const char *data,
        int expected)
{
    lib_function func = dlsym(lib, func_to_run);
    check(func != NULL,
            "Did not find %s function in the library %s: %s", func_to_run,
            lib_file, dlerror());

    int rc = func(data);
    check(rc == expected, "Function %s return %d for data: %s",
            func_to_run, rc, data);

    return 1;
error:
    return 0;
}

char *test_dlopen()
{
    lib = dlopen(lib_file, RTLD_NOW);
    mu_assert(lib != NULL, "Failed to open the library to test.");

    return NULL;
}

char *test_functions()
{
    mu_assert(check_function("print_a_message", "Hello", 0),
            "print_a_message failed.");
    mu_assert(check_function("uppercase", "Hello", 0),
            "uppercase failed.");
    mu_assert(check_function("lowercase", "Hello", 0),
            "lowercase failed.");

    return NULL;
}

char *test_failures()
{
    mu_assert(check_function("fail_on_purpose", "Hello", 1),
            "fail_on_purpose should fail.");

    return NULL;
}

char *test_dlclose()
{
    int rc = dlclose(lib);
    mu_assert(rc == 0, "Failed to close lib.");

    return NULL;
}

char *all_tests()
{
    mu_suite_start();

    mu_run_test(test_dlopen);
    mu_run_test(test_functions);
    mu_run_test(test_failures);
    mu_run_test(test_dlclose);

    return NULL;
}

RUN_TESTS(all_tests);

```

.\ex48b\statserve

.\ex48b\statserve\bin\statserve.c

```c
#include <stdio.h>
#include <lcthw/dbg.h>
#include "statserve.h"
#include "net.h"

int main(int argc, char *argv[])
{
    check(argc == 3, "USAGE: statserve host port");

    const char *host = argv[1];
    const char *port = argv[2];

    check(echo_server(host, port), "Failed to run the echo server.");

    return 0;

error:

    return 1;
}

```

.\ex48b\statserve\src\dbg.h

```c
#ifndef __dbg_h__
#define __dbg_h__

#include <stdio.h>
#include <errno.h>
#include <string.h>

#ifdef NDEBUG
#define debug(M, ...)
#else
#define debug(M, ...) fprintf(stderr, "DEBUG %s:%d: " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)
#endif

#define clean_errno() (errno == 0 ? "None" : strerror(errno))

#define log_err(M, ...) fprintf(stderr,\
        "[ERROR] (%s:%d: errno: %s) " M "\n", __FILE__, __LINE__,\
        clean_errno(), ##__VA_ARGS__)

#define log_warn(M, ...) fprintf(stderr,\
        "[WARN] (%s:%d: errno: %s) " M "\n",\
        __FILE__, __LINE__, clean_errno(), ##__VA_ARGS__)

#define log_info(M, ...) fprintf(stderr, "[INFO] (%s:%d) " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)

#define check(A, M, ...) if(!(A)) {\
    log_err(M, ##__VA_ARGS__); errno=0; goto error; }

#define sentinel(M, ...)  { log_err(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#define check_mem(A) check((A), "Out of memory.")

#define check_debug(A, M, ...) if(!(A)) { debug(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#endif

```

.\ex48b\statserve\src\net.c

```c
#include <stdlib.h>
#include <sys/select.h>
#include <stdio.h>
#include <lcthw/ringbuffer.h>
#include <lcthw/dbg.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <sys/uio.h>
#include <arpa/inet.h>
#include <netdb.h>
#include <unistd.h>
#include <fcntl.h>
#include "net.h"

struct tagbstring NL = bsStatic("\n");
struct tagbstring CRLF = bsStatic("\r\n");

int nonblock(int fd)
{
    int flags = fcntl(fd, F_GETFL, 0);
    check(flags >= 0, "Invalid flags on nonblock.");

    int rc = fcntl(fd, F_SETFL, flags | O_NONBLOCK);
    check(rc == 0, "Can't set nonblocking.");

    return 0;
error:
    return -1;
}

int client_connect(char *host, char *port)
{
    int rc = 0;
    struct addrinfo *addr = NULL;

    rc = getaddrinfo(host, port, NULL, &addr);
    check(rc == 0, "Failed to lookup %s:%s", host, port);

    int sock = socket(AF_INET, SOCK_STREAM, 0);
    check(sock >= 0, "Cannot create a socket.");

    rc = connect(sock, addr->ai_addr, addr->ai_addrlen);
    check(rc == 0, "Connect failed.");

    rc = nonblock(sock);
    check(rc == 0, "Can't set nonblocking.");

    freeaddrinfo(addr);
    return sock;

error:
    freeaddrinfo(addr);
    return -1;
}

int read_some(RingBuffer * buffer, int fd, int is_socket)
{
    int rc = 0;

    if (RingBuffer_available_data(buffer) == 0) {
        buffer->start = buffer->end = 0;
    }

    if (is_socket) {
        rc = recv(fd, RingBuffer_starts_at(buffer),
                RingBuffer_available_space(buffer), 0);
    } else {
        rc = read(fd, RingBuffer_starts_at(buffer),
                RingBuffer_available_space(buffer));
    }

    check(rc >= 0, "Failed to read from fd: %d", fd);

    RingBuffer_commit_write(buffer, rc);

    return rc;

error:
    return -1;
}

int write_some(RingBuffer * buffer, int fd, int is_socket)
{
    int rc = 0;
    bstring data = RingBuffer_get_all(buffer);

    check(data != NULL, "Failed to get from the buffer.");
    check(bfindreplace(data, &NL, &CRLF, 0) == BSTR_OK,
            "Failed to replace NL.");

    if (is_socket) {
        rc = send(fd, bdata(data), blength(data), 0);
    } else {
        rc = write(fd, bdata(data), blength(data));
    }

    check(rc == blength(data), "Failed to write everything to fd: %d.",
            fd);
    bdestroy(data);

    return rc;

error:
    return -1;
}

int attempt_listen(struct addrinfo *info)
{
    int sockfd = -1; // default fail
    int rc = -1;
    int yes = 1;

    check(info != NULL, "Invalid addrinfo.");

    // create a socket with the addrinfo
    sockfd = socket(info->ai_family, info->ai_socktype,
            info->ai_protocol);
    check_debug(sockfd != -1, "Failed to bind to address. Trying more.");

    // set the SO_REUSEADDR option on the socket
    rc = setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(int));
    check_debug(rc == 0, "Failed to set SO_REUSADDR.");

    // attempt to bind to it
    rc = bind(sockfd, info->ai_addr, info->ai_addrlen);
    check_debug(rc == 0, "Failed to find socket.");

    // finally listen with a backlog
    rc = listen(sockfd, BACKLOG);
    check_debug(rc == 0, "Failed to listen to socket.");

    return sockfd;

error:
    return -1;
}

int server_listen(const char *host, const char *port)
{
    int rc = 0;
    int sockfd = -1; // default fail value
    struct addrinfo *info = NULL;
    struct addrinfo *next_p = NULL;
    struct addrinfo addr = {
        .ai_family = AF_UNSPEC,
        .ai_socktype = SOCK_STREAM,
        .ai_flags = AI_PASSIVE
    };

    check(host != NULL, "Invalid host.");
    check(port != NULL, "Invalid port.");

    // get the address info for host and port
    rc = getaddrinfo(NULL, port, &addr, &info);
    check(rc == 0, "Failed to get address info for connect.");

    // cycle through the available list to find one
    for(next_p = info; next_p != NULL; next_p = next_p->ai_next)
    {
        // attempt to listen to each one
        sockfd = attempt_listen(next_p);
        if(sockfd != -1) break;
    }

    // either we found one and were able to listen or nothing.
    check(sockfd != -1, "All possible addresses failed.");

error: //fallthrough
    if(info) freeaddrinfo(info);
    // this gets set by the above to either -1 or valid
    return sockfd;
}

```

.\ex48b\statserve\src\net.h

```c
#ifndef _net_h
#define _net_h

#include <lcthw/ringbuffer.h>

#define BACKLOG 10

int nonblock(int fd);
int client_connect(char *host, char *port);
int read_some(RingBuffer * buffer, int fd, int is_socket);
int write_some(RingBuffer * buffer, int fd, int is_socket);
int server_listen(const char *host, const char *port);

#endif

```

.\ex48b\statserve\src\statserve.c

```c
#include <stdio.h>
#include <ctype.h>
#include <lcthw/dbg.h>
#include <unistd.h>
#include <stdlib.h>
#include <signal.h>
#include <sys/wait.h>
#include "net.h"
#include <netdb.h>

const int RB_SIZE = 1024 * 10;

void handle_sigchild(int sig) {
    sig = 0; // ignore it
    while(waitpid(-1, NULL, WNOHANG) > 0) {
    }
}

void client_handler(int client_fd)
{
    int rc = 0;
    // need a ringbuffer for the input
    RingBuffer *sock_rb = RingBuffer_create(RB_SIZE);

    // read_some in a loop
    while(read_some(sock_rb, client_fd, 1) != -1) {
        // write_it back off the ringbuffer
        if(write_some(sock_rb, client_fd, 1) == -1) {
            debug("Client closed.");
            break;
        }
    }

    // close the socket
    rc = close(client_fd);
    check(rc != -1, "Failed to close the socket.");

error: // fallthrough
    if(sock_rb) RingBuffer_destroy(sock_rb);
    exit(0); // just exit the child process
}

int echo_server(const char *host, const char *port)
{
    int rc = 0;
    struct sockaddr_in client_addr;
    socklen_t sin_size = sizeof(client_addr);
    int server_socket = 0;
    int client_fd = 0;

    struct sigaction sa = {
        .sa_handler = handle_sigchild,
        .sa_flags = SA_RESTART | SA_NOCLDSTOP
    };

    check(host != NULL, "Invalid host.");
    check(port != NULL, "Invalid port.");

    // create a sigaction that handles SIGCHLD
    sigemptyset(&sa.sa_mask);
    rc = sigaction(SIGCHLD, &sa, 0);
    check(rc != -1, "Failed to setup signal handler for child processes.");

    // listen on the given port and host
    server_socket = server_listen(host, port);
    check(server_socket >= 0, "bind to %s:%s failed.", host, port);

    while(1) {
        // accept the connection
        client_fd = accept(server_socket, (struct sockaddr *)&client_addr, &sin_size); 
        check(client_fd >= 0, "Failed to accept connection.");

        debug("Client connected.");

        rc = fork();
        if(rc == 0) {
            // child process
            close(server_socket); // don't need this
            // handle the client
            client_handler(client_fd);
        } else {
            // server process
            close(client_fd); // don't need this
        }
    }

error:  // fallthrough
    return -1;
}

```

.\ex48b\statserve\src\statserve.h

```c
#ifndef _statserve_h
#define _statserve_h

int echo_server(const char *host, const char *port);

#endif

```

.\ex48b\statserve\tests\statserve_tests.c

```c
#include "minunit.h"
#include <dlfcn.h>
#include "statserve.h"

char *test_dummy()
{
    return NULL;
}

char *all_tests()
{
    mu_suite_start();

    mu_run_test(test_dummy);

    return NULL;
}

RUN_TESTS(all_tests);

```

Solution

The Plan

Show you how I solved the *statserve* project.

The Purpose

Watch me solve the first project quickly, then review the code.

The Setup

First I need to install liblcthw since I'll be using that.

Then I make the project skeleton and get something, anything going.

The Server

Then I just get it accepting a connection.

The Echo

Then I decided to just make it echo back what I type.

The Final Code

<file_sep>+++

title = "Java Note - 3: Path and Files"
description="Some good part of Java 7 - Path and Files API"
+++


## Prerequisites

>*Java 7+*

## Good stuff from not shiny Java 7

If you are planning to refactor your code, please give a second thought. It is time to dump to try these new features. When the Java 7 was released, I was kind of disappointed without lambda, jigsaw as most developers, but when I tried new Path, Files API, I found that is great improvement. The enhancement of this new IO is really useful. It save so much effort for Java developer. 

To be hoenst, before Java 7, Coding file manipulation in Java is very headache task. I say "headache" it doesn't mean it is difficult. Just comparing with other program lanugage, you had to take much more effort to take care of the boilerplate, and all are tedious job. That is why sometimes I prefer cmd in Window or bash in Linux to complete the task instead of using Java to handle file manipulation. Now I think I can refactor old file manipulation coding and make it much more elegant. 


### Better file visitor implementation

Following is simple customizaed file visitor which has been the part of my old util.

```java
import java.io.IOException;
import java.nio.file.FileVisitResult;
import java.nio.file.Path;
import java.nio.file.SimpleFileVisitor;
import java.nio.file.attribute.BasicFileAttributes;

public class CustomFileVisitor extends SimpleFileVisitor<Path> {
    @Override
    public FileVisitResult postVisitDirectory(Path dir , IOException arg1) throws IOException {
        System.out.println( "post visit dir :  "+  dir );
        return FileVisitResult.CONTINUE;
        }

        @Override
    public FileVisitResult preVisitDirectory(Path dir , IOException arg1) throws IOException {
        System.out.println( "post visit dir :  "+  dir );
        return FileVisitResult.CONTINUE;
        }

    @Override
    public FileVisitResult visitFile(Path file, BasicFileAttributes attr)
            throws IOException {
         if ( attr.isSymbolicLink() )
          { System.out.println( " symbolic link  :  "+  file  );
          
          }else if (  attr.isSymbolicLink() ){              
              System.out.println( " regular file : "+ file );
          }
        return FileVisitResult.CONTINUE;
    }
     
    @Override
    public FileVisitResult visitFileFailed(Path file, IOException exc)
            throws IOException {        
         System.err.println( exc.getMessage());
        return FileVisitResult.CONTINUE;
    }
}
```

To use this customized  is so easy. Just 3 lines coding you can test it by yourself. 
```
        CustomFileVisitor fileVisitor = new CustomFileVisitor();
        Path path = Paths.get("TestDir");
        Files.walkFileTree(path, fileVisitor);
```

### ARM 

Automatic resource management is another attractive features of Java 7 and project coin. As name itself implies that now JVM is going to be handling all the external resource and make programmer free to bother about resource management, especially for people like me miss the `using` statement in C#. Sometimes I wonder why Java is such stubborn not to learn some good features from C#. As we know, C# comes after Java and copies most concept at the early stage, but it really pushed Object Oriented Concept (OOC) to a new level and inspired Java world a lot with its many good feature. I really hope someday I can code in Java as simple as C#. Wise men learn by other men's mistakes; fools by their own. 

In the past, java programmers use any external resources like file, printer or any devices to close after my program execution complete. Normally we close the resources which we have open in beginning of our program or we decide that if program finish normally how to manage the resource or if our program finish abnormally how to close the resource. Following are comparison of old and new style. 

*Snippet of old style*

```java
FileInputStream exchangeCurrencyReader= null;
FileOutputStream exchangeCurrencyWriter = null;
try {
    exchangeCurrencyReader = new FileInputStream("AUDvsUSD.txt");
    exchangeCurrencyWriter = new FileOutputStream("AUDvsUSD.txt");
    int var;
    while (var = exchangeCurrencyReader.read()) != -1)
        exchangeCurrencyWriter.write(var);
} 
finally {
    if (exchangeCurrencyReader!= null)
        exchangeCurrencyReader.close();
    if (exchangeCurrencyWriter!= null)
        exchangeCurrencyWriter.close();
}
```

*Code in Java 7*

```java
try ( FileInputStream exchangeCurrencyReader = new FileInputStream("AUDvsUSD.txt");
    FileOutputStream exchangeCurrencyWriter = new FileOutputStream("AUDvsUSD.txt")){
      int var;
      while((var= exchangeCurrencyReader.read()) != -1 )
            exchangeCurrencyWriter.write();
}

```

In the code above we have declare two file stream one is input file we are reading from one file and writing to another file. After the whole process both streams will be closed automatically either the code has been executed properly or not. Both exchangeCurrencyReader.close() and exchangeCurrencyWriter.close() methods will be called automatically which is the best part of ARM. We should not miss good part from Java 7. 


### New file change monitor service

After some homework for new features of Java 7, I am tring to use file watch serviice from Java 7 to replace old file monitor program. It is great and quite simple to use. I have updated to production. 

Usually most Java based system somehow needs such monitor function, there will be separate process or thread to run this service, and there should be a call back handler triggered by this service. Everytime there is any file amended, the service will trigger the call back handler to complete some tasks. 

Following is sample of file watching service. I removed call back part which is relevant to the business. 

```java
import java.nio.file.FileSystems;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardWatchEventKinds;
import java.nio.file.WatchEvent;
import java.nio.file.WatchKey;
import java.nio.file.WatchService;
import java.util.HashMap;
import java.util.Map;

public class FileWatchService {

    public static void watchFileUpdate() {
        try (WatchService service = FileSystems.getDefault().newWatchService()) {
            Map<WatchKey, Path> eventMap = new HashMap<>();
            Path dir = Paths.get("TestDir");

            eventMap.put(dir.register(service, StandardWatchEventKinds.ENTRY_MODIFY),dir);

            WatchKey key;
            do {
                key = service.take();
                Path eventPath = eventMap.get(key);
                for (WatchEvent<?> event : key.pollEvents()) {
                    WatchEvent.Kind<?> kind = event.kind();
                    Path path = (Path) event.context();
                    System.out.println(eventPath + " : " + kind + "  : " + path);
                }
            } while (key.reset());
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
    
    public static void main (String [] args ){
       watchFileUpdate();
    }
}
```

The sample above shows one kind of events. Actually if you check the API doc, there are another two kinds of event. One is  StandardWatchEventKinds.ENTRY\_CREATE , the other is  StandardWatchEventKinds.ENTRY\_DELETE. These events cover almostly all business requirements.   




































































<file_sep>+++
title = "F# Task"
description = "F# Computation - Task expressions"
weight = 13
+++

## Tasks expressions

Asynchronous code is normally authored using async expressions. Using task expressions is preferred when interoperating extensively with .NET libraries that create or consume .NET tasks. Task expressions can also improve performance and the debugging experience. However, task expressions come with some limitations.


`task { expression }`

The task is started immediately after this code is executed and runs on the current thread until its first asynchronous operation is performed (for example, an asynchronous sleep, asynchronous I/O, or other primitive asynchronous operation). The type of the expression is `Task<'T>`, where `'T` is the type returned by the expression when the return keyword is used.


### Binding by using let!

In a task expression, some expressions and operations are synchronous, and some are asynchronous. When you await the result of an asynchronous operation, instead of an ordinary let binding, you use let!. The effect of let! is to enable execution to continue on other computations or threads as the computation is being performed. After the right side of the let! binding returns, the rest of the task resumes execution.

The following code shows the difference between let and let!. The line of code that uses let just creates a task as an object that you can await later by using, for example, task.Wait() or task.Result. The line of code that uses let! starts the task and awaits its result.

```fsharp
// let just stores the result as a task.
let (result1 : Task<int>) = stream.ReadAsync(buffer, offset, count, cancellationToken)
// let! completes the asynchronous operation and returns the data.
let! (result2 : int)  = stream.ReadAsync(buffer, offset, count, cancellationToken)
```

F# `task { }` expressions can await the following kinds of asynchronous operations:

- .NET tasks, Task<TResult> and the non-generic Task.
- .NET value tasks, ValueTask<TResult> and the non-generic ValueTask.
- F# async computations Async<T>.
- Any object following the "GetAwaiter" pattern specified in F# RFC FS-1097.

### return expressions

Within task expressions, `return` expr is used to return the result of a task.

### return! expressions

Within task expressions, `return!` expr is used to return the result of another task. It is equivalent to using let! and then immediately returning the result.
Control flow

Task expressions can include the control-flow constructs `for .. in .. do, while .. do, try .. with .., try .. finally .., if .. then .. else, and if .. then ...` These may in turn include further task constructs, except for the with and finally handlers, which execute synchronously. If you need an asynchronous `try .. finally ..`, use a use binding in combination with an object of type `IAsyncDisposable`.
`use` and `use!` bindings

Within task expressions, use bindings can bind to values of type `IDisposable` or `IAsyncDisposable`. For the latter, the disposal cleanup operation is executed asynchronously.

In addition to let!, you can use  `use!` to perform asynchronous bindings. The difference between let! and  `use!` is the same as the difference between let and use. For use!, the object is disposed of at the close of the current scope. Note that in F# 6,  `use!` does not allow a value to be initialized to null, even though use does.


### Value Tasks

Value tasks are structs used to avoid allocations in task-based programming. A value task is an ephemeral value that's turned into a real task by using `.AsTask()`.

To create a value task from a task expression, `use |> ValueTask<ReturnType> or |> ValueTask`. For example:
F#

```fsharp
let makeTask() =
    task { return 1 }

makeTask() |> ValueTask<int>
```


### Adding cancellation tokens and cancellation checks

Unlike F# async expressions, task expressions do not implicitly pass a cancellation token and don't implicitly perform cancellation checks. If your code requires a cancellation token, you should specify the cancellation token as a parameter. 

```fsharp
open System.Threading

let someTaskCode (cancellationToken: CancellationToken) =
    task {
        cancellationToken.ThrowIfCancellationRequested()
        printfn $"continuing..."
    }
```


If you intend to correctly make your code cancelable, carefully check that you pass the cancellation token through to all .NET library operations that support cancellation. For example, Stream.ReadAsync has multiple overloads, one of which accepts a cancellation token. If you do not use this overload, that specific asynchronous read operation will not be cancelable.


### Background tasks

By default, .NET tasks are scheduled using `SynchronizationContext.Current` if present. This allows tasks to serve as cooperative, interleaved agents executing on a user interface thread without blocking the UI. If not present, task continuations are scheduled to the .NET thread pool.

In practice, it's often desirable that library code that generates tasks ignores the synchronization context and instead always switches to the .NET thread pool, if necessary. You can achieve this using `backgroundTask { }`:

`backgroundTask { expression }`

A background task ignores any `SynchronizationContext.Current` in the following sense: if started on a thread with non-null `SynchronizationContext.Current`, it switches to a background thread in the thread pool using Task.Run. If started on a thread with null `SynchronizationContext.Current`, it executes on that same thread.

### Note

In practice, this means that calls to `ConfigureAwait(false)` are not typically needed in F# task code. Instead, tasks that are intended to run in the background should be authored using `backgroundTask { ... }`. Any outer task binding to a background task will resynchronize to the `SynchronizationContext.Current` on completion of the background task.

### Limitations of tasks regarding tailcalls

Unlike F# async expressions, task expressions do not support tailcalls. That is, when `return!` is executed, the current task is registered as awaiting the task whose result is being returned. This means that recursive functions and methods implemented using task expressions may create unbounded chains of tasks, and these may use unbounded stack or heap. For example, consider the following code:
F#

```fsharp
let rec taskLoopBad (count: int) : Task<string> =
    task {
        if count = 0 then
            return "done!"
        else
            printfn $"looping..., count = {count}"
            return! taskLoopBad (count-1)
    }

let t = taskLoopBad 10000000
t.Wait()
```

This coding style should not be used with task expressions—it will create a chain of 10000000 tasks and cause a `StackOverflowException`. If an asynchronous operation is added on each loop invocation, the code will use an essentially unbounded heap. Consider switching this code to use an explicit loop, for example:


```fsharp
let taskLoopGood (count: int) : Task<string> =
    task {
        for i in count .. 1 do
            printfn $"looping... count = {count}"
        return "done!"
    }

let t = taskLoopGood 10000000
t.Wait()
```

If asynchronous tailcalls are required, use an F# async expression, which does support tailcalls. For example:

```fsharp
let rec asyncLoopGood (count: int) =
    async {
        if count = 0 then
            return "done!"
        else
            printfn $"looping..., count = {count}"
            return! asyncLoopGood (count-1)
    }

let t = asyncLoopGood 1000000 |> Async.StartAsTask
t.Wait()
```
<file_sep>+++
title = "Adv Bash - 3"
description = "Reference Cards - Files"
+++


### Operators: Files

Operator	|Tests Whether|Operator	|Tests Whether
:--|------|-----|------
| -e	|File exists	 	| -s	| File is not zero size
| -f	|File is a regular file	 	 	 
| -d	|File is a directory	 	| -r	| File has read permission
| -h	|File is a symbolic link	 	| -w	| File has write permission
| -L	|File is a symbolic link	 	| -x	| File has execute permission
| -b	|File is a block device	 	 	 ||
| -c	|File is a character device	 	| -g	| sgid flag set
| -p	|File is a pipe	 	| -u	| suid flag set 
| -S	|File is a socket	 	| -k	| "sticky bit" set
| -t	|File is associated with a terminal	 ||	 	 
| -N	|File modified since it was last read	| 	F1 -nt F2	| File F1 is newer than F2 *
| -O	|You own the file	 |	F1 -ot F2	| File F1 is older than F2 *
| -G	|Group id of file same as yours	 |	F1 -ef F2	| Files F1 and F2 are hard links to the same file * 
|  !	| NOT (inverts sense of above tests) ||	 	 	 




### Sample 


```

function is_symbolic_link(){
    echo "Is the file $1  a symbolic link?";

    if  [ -f $1 ] && [ -h $1 ]; 
    then
        echo 'true' ; 
    else 
        echo 'false'; 
    fi;
}

function is_file_empty () {

    echo "Is the file $1  empty?";

    if  [ -f $1 ] && [ ! -s $1 ]; 
    then
        echo 'true' ; 
    else 
        echo 'false'; 
    fi;
}
```












<file_sep>+++
title = "AWS: ECS - 1"
description = "Getting started with Fargate"
draft="true"
+++


## ECS 

Amazon Elastic Container Service (Amazon ECS) is a highly scalable, fast, container management service that makes it easy to run, stop, and manage Docker containers on a cluster. You can host your cluster on a serverless infrastructure that is managed by Amazon ECS by launching your services or tasks using the Fargate launch type. For more control you can host your tasks on a cluster of Amazon Elastic Compute Cloud (Amazon EC2) instances that you manage by using the EC2 launch type. 

Amazon ECS lets you launch and stop container-based applications with simple API calls, allows you to get the state of your cluster from a centralized service, and gives you access to many familiar Amazon EC2 features.

Amazon ECS can be used to create a consistent deployment and build experience, manage, and scale batch and Extract-Transform-Load (ETL) workloads, and build sophisticated application architectures on a microservices model. 



### Installation 

* MacOS

        sudo curl -o /usr/local/bin/ecs-cli https://amazon-ecs-cli.s3.amazonaws.com/ecs-cli-darwin-amd64-latest

* Linux 

        sudo curl -o /usr/local/bin/ecs-cli https://amazon-ecs-cli.s3.amazonaws.com/ecs-cli-linux-amd64-latest
    

### Launcth with Fargate 


* Create a cluster configuration
  
        ecs-cli configure --cluster pg-far \
            --default-launch-type FARGATE \
            --config-name pg-far --region ap-southeast-2


* Create a CLI profile 

        ecs-cli configure profile \
            --access-key AWS_ACCESS_KEY_ID \
            --secret-key AWS_SECRET_ACCESS_KEY \
            --profile-name pg-far-profile



* Create a Cluster 
  
      ecs-cli up --cluster-config pg-far \
        --ecs-profile pg-far-profile


* Configure the Security Group. 

        # VPC_ID is from command above
        aws ec2 describe-security-groups \
            --filters Name=vpc-id,Values=<VPC_ID> \
            --region ap-southeast-2

* Add a rule to allow inbound access on port 80.

        aws ec2 authorize-security-group-ingress \
            --group-id <security_group_id> \
            --protocol tcp --port 80 \
            --cidr 0.0.0.0/0 --region ap-southeast-2

* Create a Compose File - docker-compose.yml

```yaml
version: '3'
services:
web:
    image: amazon/amazon-ecs-sample
    ports:
    - "80:80"
    logging:
    driver: awslogs
    options: 
        awslogs-group: pg-far
        awslogs-region: ap-southeast-2
        awslogs-stream-prefix: web
```

* Create a parameters specific file - ecs-params.yml


```yaml
version: 1
task_definition:
  task_execution_role: ecsTaskExecutionRole
  ecs_network_mode: awsvpc
  task_size:
    mem_limit: 0.5GB
    cpu_limit: 256
run_params:
  network_configuration:
    awsvpc_configuration:
      subnets:
        - "subnet ID 1"
        - "subnet ID 2"
      security_groups:
        - "security group ID"
      assign_public_ip: ENABLED
```

* Deploy the Compose File to a Cluster

        ecs-cli compose --project-name pg-far \
            service up --create-log-groups \
            --cluster-config pg-far \
            --ecs-profile pg-far-profile

* View the container

        ecs-cli compose --project-name pg-far \
            service ps --cluster-config pg-far \
            --ecs-profile pg-far-profile

* View the log

        ecs-cli logs --task-id 0c2862e6e39e4eff92ca3e4f843c5b9a \
            --follow --cluster-config pg-far \
            --ecs-profile pg-far-profile


<file_sep>+++
title = "F# Computations 1"
description = "F# Computation expressions "
weight = 10
+++

## Computation expressions

Computation expressions in F# provide a convenient syntax for writing computations that can be sequenced and combined using control flow constructs and bindings. Depending on the kind of computation expression, they can be thought of as a way to express monads, monoids, monad transformers, and applicative functors. However, unlike other languages (such as do-notation in Haskell), they are not tied to a single abstraction, and do not rely on macros or other forms of metaprogramming to accomplish a convenient and context-sensitive syntax.

### Syntax overview

```fsharp
builder-expr { cexper }

expr { let! ... }
expr { do! ... }
expr { yield ... }
expr { yield! ... }
expr { return ... }
expr { return! ... }
expr { match! ... }
```

#### let!

The let! keyword binds the result of a call to another computation expression to a name. let! is defined by the Bind(x, f) member on the builder type.

#### do!

The do! keyword is for calling a computation expression that returns a unit-like type (defined by the Zero member on the builder)





### Built-in computation expressions

The F# core library defines four built-in computation expressions: Sequence Expressions, Async expressions, Task expressions, and Query Expressions.


### Custom computation expression

Every computation expression is backed by a builder type. The builder type defines the operations that are available for the computation expression. Following table shows how to create a custom computation expression.

Method | Typical signature(s) | Description
---|---|---
Bind | M<'T> * ('T -> M<'U>) -> M<'U> | Called for let! and do! in computation expressions.
Delay | (unit -> M<'T>) -> Delayed<'T> | Wraps a computation expression as a function. Delayed<'T> can be any type, commonly M<'T> or unit -> M<'T> are used. The default implementation returns a M<'T>.
Return | 'T -> M<'T> | Called for return in computation expressions.
ReturnFrom | M<'T> -> M<'T> | Called for return! in computation expressions.
Run | Delayed<'T> -> M<'T> or M<'T> -> 'T | Executes a computation expression.
Combine | M<'T> * Delayed<'T> -> M<'T> or M<unit> * M<'T> -> M<'T> | Called for sequencing in computation expressions.
For | seq<'T> * ('T -> M<'U>) -> M<'U> or seq<'T> * ('T -> M<'U>) -> seq<M<'U>> | Called for for...do expressions in computation expressions.
TryFinally | Delayed<'T> * (unit -> unit) -> M<'T> | Called for try...finally expressions in computation expressions.
TryWith | Delayed<'T> * (exn -> M<'T>) -> M<'T> | Called for try...with expressions in computation expressions.
Using | 'T * ('T -> M<'U>) -> M<'U> when 'T :> IDisposable | Called for use bindings in computation expressions.
While | (unit -> bool) * Delayed<'T> -> M<'T>or (unit -> bool) * Delayed<unit> -> M<unit> | Called for while...do expressions in computation expressions.
Yield | 'T -> M<'T> | Called for yield expressions in computation expressions.
YieldFrom | M<'T> -> M<'T> | Called for yield! expressions in computation expressions.
Zero | unit -> M<'T> | Called for empty else branches of if...then expressions in computation expressions.
Quote | Quotations.Expr<'T> -> Quotations.Expr<'T> | Indicates that the computation expression is passed to the Run member as a quotation. It translates all instances of a computation into a quotation.


The following code example shows a computation expression that encapsulates a computation as a series of steps that can be evaluated one step at a time. A discriminated union type, OkOrException, encodes the error state of the expression as evaluated so far. This code demonstrates several typical patterns that you can use in your computation expressions, such as boilerplate implementations of some of the builder methods.

```fsharp   
/// Represents computations that can be run step by step
type Eventually<'T> =
    | Done of 'T
    | NotYetDone of (unit -> Eventually<'T>)

module Eventually =

    /// Bind a computation using 'func'.
    let rec bind func expr =
        match expr with
        | Done value -> func value
        | NotYetDone work -> NotYetDone (fun () -> bind func (work()))

    /// Return the final value
    let result value = Done value

    /// The catch for the computations. Stitch try/with throughout
    /// the computation, and return the overall result as an OkOrException.
    let rec catch expr =
        match expr with
        | Done value -> result (Ok value)
        | NotYetDone work ->
            NotYetDone (fun () ->
                let res = try Ok(work()) with | exn -> Error exn
                match res with
                | Ok cont -> catch cont // note, a tailcall
                | Error exn -> result (Error exn))

    /// The delay operator.
    let delay func = NotYetDone (fun () -> func())

    /// The stepping action for the computations.
    let step expr =
        match expr with
        | Done _ -> expr
        | NotYetDone func -> func ()

    /// The tryFinally operator.
    /// This is boilerplate in terms of "result", "catch", and "bind".
    let tryFinally expr compensation =
        catch (expr)
        |> bind (fun res ->
            compensation();
            match res with
            | Ok value -> result value
            | Error exn -> raise exn)

    /// The tryWith operator.
    /// This is boilerplate in terms of "result", "catch", and "bind".
    let tryWith exn handler =
        catch exn
        |> bind (function Ok value -> result value | Error exn -> handler exn)

    /// The whileLoop operator.
    /// This is boilerplate in terms of "result" and "bind".
    let rec whileLoop pred body =
        if pred() then body |> bind (fun _ -> whileLoop pred body)
        else result ()

    /// The sequential composition operator.
    /// This is boilerplate in terms of "result" and "bind".
    let combine expr1 expr2 =
        expr1 |> bind (fun () -> expr2)

    /// The using operator.
    /// This is boilerplate in terms of "tryFinally" and "Dispose".
    let using (resource: #System.IDisposable) func =
        tryFinally (func resource) (fun () -> resource.Dispose())

    /// The forLoop operator.
    /// This is boilerplate in terms of "catch", "result", and "bind".
    let forLoop (collection:seq<_>) func =
        let ie = collection.GetEnumerator()
        tryFinally
            (whileLoop
                (fun () -> ie.MoveNext())
                (delay (fun () -> let value = ie.Current in func value)))
            (fun () -> ie.Dispose())

/// The builder class.
type EventuallyBuilder() =
    member x.Bind(comp, func) = Eventually.bind func comp
    member x.Return(value) = Eventually.result value
    member x.ReturnFrom(value) = value
    member x.Combine(expr1, expr2) = Eventually.combine expr1 expr2
    member x.Delay(func) = Eventually.delay func
    member x.Zero() = Eventually.result ()
    member x.TryWith(expr, handler) = Eventually.tryWith expr handler
    member x.TryFinally(expr, compensation) = Eventually.tryFinally expr compensation
    member x.For(coll:seq<_>, func) = Eventually.forLoop coll func
    member x.Using(resource, expr) = Eventually.using resource expr

let eventually = new EventuallyBuilder()

let comp =
    eventually {
        for x in 1..2 do
            printfn $" x = %d{x}"
        return 3 + 4
    }

/// Try the remaining lines in F# interactive to see how this
/// computation expression works in practice.
let step x = Eventually.step x

// returns "NotYetDone <closure>"
comp |> step

// prints "x = 1"
// returns "NotYetDone <closure>"
comp |> step |> step

// prints "x = 1"
// prints "x = 2"
// returns "Done 7"
comp |> step |> step |> step |> step

```

### Translation of expression

Expression|Translation
-----|------
{ let binding in cexpr }|let binding in {\| cexpr \|}
{ let! pattern = expr in cexpr }|builder.Bind(expr, (fun pattern -> {\| cexpr \|}))
{ do! expr in cexpr }|builder.Bind(expr, (fun () -> {\| cexpr \|}))
{ yield expr }|builder.Yield(expr)
{ yield! expr }|builder.YieldFrom(expr)
{ return expr }|builder.Return(expr)
{ return! expr }|builder.ReturnFrom(expr)
{ use pattern = expr in cexpr }|builder.Using(expr, (fun pattern -> {\| cexpr \|}))
{ use! value = expr in cexpr }|builder.Bind(expr, (fun value -> builder.Using(value, (fun value -> { cexpr }))))
{ if expr then cexpr0 |}|if expr then { cexpr0 } else builder.Zero()
{ if expr then cexpr0 else cexpr1 |}|if expr then { cexpr0 } else { cexpr1 }
{ match expr with \| pattern_i -> cexpr_i }|match expr with \| pattern_i -> { cexpr_i }
{ for pattern in expr do cexpr }|builder.For(enumeration, (fun pattern -> { cexpr }))
{ for identifier = expr1 to expr2 do cexpr }|builder.For(enumeration, (fun identifier -> { cexpr }))
{ while expr do cexpr }|builder.While(fun () -> expr, builder.Delay({ cexpr }))
{ try cexpr with \| pattern_i -> expr_i }|builder.TryWith(builder.Delay({ cexpr }), (fun value -> match value with \| pattern_i -> expr_i \| exn -> System.Runtime.ExceptionServices.ExceptionDispatchInfo.Capture(exn).Throw())))
{ try cexpr finally expr }|builder.TryFinally(builder.Delay( { cexpr }), (fun () -> expr))
{ cexpr1; cexpr2 }|builder.Combine({ cexpr1 }, { cexpr2 })
{ other-expr; cexpr }|expr; { cexpr }
{ other-expr }|expr; builder.Zero()












<file_sep>+++

title = "Exception & Assertion"
description = "Exception & Assertion"
weight=7
+++


## Exception & Assertion

* Avoid bad practices in Python exception handling.
* Always specify an exception type with except, but don't be too general.    
* Don't Use Assertions for checking arguments
* EXamples
    * lookup exception

        ```python
        def lookups():
            s = [1, 4, 6]
            try:
                item = s[5]
            except LookupError:
                print("Handled IndexError")

            d = dict(a=65, b=66, c=67)
            try:
                value = d['x']
            except LookupError:
                print("Handled KeyError")

        if __name__ == '__main__':
            lookups()

        ## test result 
        ## Handled IndexError
        ## Handled KeyError
        ```

    * unicode exception

        ```python
        def unicode_exception():
        try:
            b'\x81'.decode('utf-8')
        except UnicodeError as e:
            print(e)
            print("encoding:", e.encoding)
            print("reason:", e.reason)
            print("object:", e.object)
            print("start:", e.start)
            print("end", e.end)

        if __name__ == '__main__':
            unicode_exception()

        ## test result
        ## 'utf-8' codec can't decode byte 0x81 in position 0: invalid start byte
        ## encoding: utf-8
        ## reason: invalid start byte
        ## object: b'\x81'
        ## start: 0
        ## end 1        
        ```

    * customized exception with parameters

        ```python
        import sys
        import io


        class TriangleError(Exception):

            def __init__(self, text, sides):
                super().__init__(text)
                self._sides = tuple(sides)

            @property
            def sides(self):
                return self._sides

            def __str__(self):
                return "'{}' for sides {}".format(self.args[0], self._sides)

            def __repr__(self):
                return "TriangleError({!r}, {!r}".format(self.args[0], self._sides)


        def triangle_area(a, b, c):
            sides = sorted((a, b, c))
            if sides[2] > sides[0] + sides[1]:
                raise TriangleError("Illegal triangle", sides)
            p = (a + b + c) / 2
            a = math.sqrt(p * (p - a) * (p - b) * (p - c))
            return a


        def triangle_exception():
            try:
                a = triangle_area(3, 4, 10)
                print(a)
            except TriangleError as e:
                try:
                    print(e, file=sys.stdin)
                except io.UnsupportedOperation as f:
                    print(e)
                    print(f)
                    print(f.__context__ is e)

        if __name__ == '__main__':
            triangle_exception()

        ## test result
        ## 'Illegal triangle' for sides (3, 4, 10)
        ## not writable
        ## True        
        ```


    * chaining & trackback

        ```python
        import math
        import traceback

        class InclinationError(Exception):
            pass


        def inclination(dx, dy):
            try:
                return math.degrees(math.atan(dy / dx))
            except ZeroDivisionError as e:
                raise InclinationError("Slope cannot be vertical") from e


        def traceback_inclination():
            try:
                inclination(0, 5)
            except InclinationError as e:
                print(e.__traceback__)
                traceback.print_tb(e.__traceback__)
                s = traceback.format_tb(e.__traceback__)
                print(s)

        if __name__ == '__main__':
            traceback_inclination()
            print("Done.")

        ## test result

        #<traceback object at 0x000000BE3B4F5108>
        ##  File "/path/to/your_project/__main__.py", line 190, in traceback_inclination
        ##    inclination(0, 5)
        ##  File "/path/to/your_project/__main__.py", line 185, in inclination
        ##    raise InclinationError("Slope cannot be vertical") from e
        #['  File "/path/to/your_project/__main__.py", line 190, 
        ## in traceback_inclination\n    
        ## inclination(0,## 5)\n',
        ##  '  File "/path/to/your_project/__main__.py", line 185, 
        ##  in inclination\n    
        ##  raise 
        ## InclinationError("Slope cannot be vertical") from e\n']

        ```

    * assertion & exception

        ```python
        from pprint import pprint as pp

        def wrap(text, line_length):
            """Wrap a string to a specified line length.

            Args:
                text: The string to wrap.
                line_length: The line length in characters.

            Returns:
                A wrapped string.

            Raises:
                ValueError: If line_length is not positive.
            """
            if line_length < 1:
                raise ValueError("line_length {} is not positive".format(line_length))

            words = text.split()

            if max(map(len, words)) > line_length:
                raise ValueError("line_length must be at least as long as the longest word")

            lines_of_words = []
            current_line_length = line_length
            for word in words:
                if current_line_length + len(word) > line_length:
                    lines_of_words.append([])  ## new line
                    current_line_length = 0
                lines_of_words[-1].append(word)
                current_line_length += len(word) + len(' ')
            lines = [' '.join(line_of_words) for line_of_words in lines_of_words]
            result = '\n'.join(lines)
            assert all(len(line) <= line_length for line in result.splitlines())
            return result

        wealth_of_nations = "The annual labour of every nation is the fund which or" \
        "iginally supplies it with all the necessaries and conveniencies of life wh" \
        "ich it annually consumes, and which consist always either in the immediate" \
        " produce of that labour, or in what is purchased with that produce from ot" \
        "her nations. According, therefore, as this produce, or what is purchased w" \
        "ith it, bears a greater or smaller proportion to the number of those who a" \
        "re to consume it, the nation will be better or worse supplied with all the" \
        " necessaries and conveniencies for which it has occasion."

        if __name__ == "__main__":
            pp(wrap( wealth_of_nations, 40))

        ## test result
        ## ('The annual labour of every nation is the\n'
        ##  'fund which originally supplies it with\n'
        ##  'all the necessaries and conveniencies of\n'
        ##  'life which it annually consumes, and\n'
        ##  'which consist always either in the\n'
        ##  'immediate produce of that labour, or in\n'
        ##  'what is purchased with that produce from\n'
        ##  'other nations. According, therefore, as\n'
        ##  'this produce, or what is purchased with\n'
        ##  'it, bears a greater or smaller\n'
        ##  'proportion to the number of those who\n'
        ##  'are to consume it, the nation will be\n'
        ##  'better or worse supplied with all the\n'
        ##  'necessaries and conveniencies for which\n'
        ##  'it has occasion.')


        ```
    
<file_sep>+++
title = "JS & ES Note - 2"
description="The equal operator doesn't always mean equivalent"
+++


## The equal operator doesn't always mean equivalent

> Have you got confused by the equal or not equal expression in the JavaScript? I will say you are definitely not the only one. Even many senior developers come from back-end programming background, they all scratch the head to find out why the equal or not-equal expression doesn't work as they expect. The truth is those expression are really different from other programming language. 



### "===" is not the same as "=="


* JavaScript has both strict and type–converting comparisons. A strict comparison (e.g., ===) is only true if the operands are of the same type and the contents match. The more commonly-used abstract comparison (e.g. ==) converts the operands to the same type before making the comparison.

* The equality operator (==) converts the operands if they are not of the same type, then applies strict comparison. If both operands are objects, then JavaScript compares internal references which are equal when operands refer to the same object in memory.

* The identity operator (===) returns true if the operands are strictly equal.

### "!=" is not the same as "!=="

* The inequality operator (!=) returns true if the operands are not equal. If the two operands are not of the same type, JavaScript attempts to convert the operands to an appropriate type for the comparison.

* The non-identity operator (!==) returns true if the operands are not equal and/or not of the same type.


### Follow are samples to show difference

> Try to test yourself before checking the answers

  
#### Sample 1


```js

console.log( " 1==true : ", 1==true)

console.log( " ''==true : ", ''==true)

console.log( " '1'==true : ", '1'==true)

console.log( " \"1\"==true : ", "1"==true)

console.log( " {}==true : ", [{}]==true)

console.log( " []==true : ", ['1']==true)

```

#### Answer of Sample 1

```js

console.log( " 1==true : ", 1==true)      // 1==true :  true
console.log( " ''==true : ", ''==true)    // ''==true :  false
console.log( " '1'==true : ", '1'==true)  // '1'==true :  true
console.log( " \"1\"==true : ", "1"==true)// "1"==true :  true
console.log( " {}==true : ", [{}]==true)  // {}==true :  false
console.log( " ['1']==true : ", ['1']==true) // []==true :  true

```

#### Sample 2



```js
console.log( " 0==false : ", 0==false)
console.log( " 1==false : ", 1==false)
console.log( " ''==false : ", ''==false)
console.log( " '1'==false : ", '1'==false)
console.log( " \"\"==false : ", ""==false)
console.log( " {}==false : ", {}==false)
console.log( " []==false : ", []==false)
console.log( " ['0']==false : ", ['0']==false)

```

#### Answer of Sample 2

```js

console.log( " 0==false : ", 0==false)    // 0==false :  true 
console.log( " 1==false : ", 1==false)    // 1==false :  false 
console.log( " ''==false : ", ''==false)  // ''==false :  true
console.log( " '1'==false : ", '1'==false)// '1'==false :  false
console.log( " \"\"==false : ", ""==false)// ""==false :  true
console.log( " {}==false : ", {}==false)  // {}==false :  false 
console.log( " []==false : ", []==false)  // []==false :  true 
console.log( " ['0']==false : ", ['0']==false)  // ['0']==false :  true 
```


#### Sample 3

```js
console.log( " null==false : ", null==false)

console.log( " undefined==false : ", undefined==false)

console.log( " undefined==null : ", null==undefined)
```


#### Answer of Sample 3


```js
console.log( " null==false : ", null==false) 
// null==false: false

console.log( " undefined==false : ", undefined==false) 
// undefined==false: false

console.log( " undefined==null : ", null==undefined) 
// undefined==null: true
```


### Tricks to compare arrays of numbers

* Compare two arrays to sure both contain the same numbers

```js
const arr1= [1,2.20,-3.5]

const arr2 =  [1.0,-3.5,2.2]

console.log(  " arr1 = arr2 ? ", obj1.sort().toString()===obj2.sort().toString())

```


<file_sep>+++
title = "F# Active Patterns"
description = "F# Active Patterns"
weight = 9
+++


## Active Patterns

Active patterns enable you to define named partitions that subdivide input data, so that you can use these names in a pattern matching expression just as you would for a discriminated union. You can use active patterns to decompose data in a customized manner for each partition.


### Syntax

```fsharp
// Active pattern of one choice.
let (|identifier|) [arguments] valueToMatch = expression

// Active Pattern with multiple choices.
// Uses a FSharp.Core.Choice<_,...,_> based on the number of case names. In F#, the limitation n <= 7 applies.
let (|identifier1|identifier2|...|) valueToMatch = expression

// Partial active pattern definition.
// Uses a FSharp.Core.option<_> to represent if the type is satisfied at the call site.
let (|identifier|_|) [arguments] valueToMatch = expression
```


### Examples


- Multiple choices

```fsharp
let (|Even|Odd|) input = if input % 2 = 0 then Even else Odd
let TestNumber input =
   match input with
   | Even -> printfn "%d is even" input
   | Odd -> printfn "%d is odd" input

TestNumber 7
TestNumber 11
TestNumber 32
// output
// 7 is odd
// 11 is odd
// 32 is even

```

- Decompose data types

```fsharp

open System.Drawing

let (|RGB|) (col : System.Drawing.Color) =
     ( col.R, col.G, col.B )

let (|HSB|) (col : System.Drawing.Color) =
   ( col.GetHue(), col.GetSaturation(), col.GetBrightness() )

let printRGB (col: System.Drawing.Color) =
   match col with
   | RGB(r, g, b) -> printfn " Red: %d Green: %d Blue: %d" r g b

let printHSB (col: System.Drawing.Color) =
   match col with
   | HSB(h, s, b) -> printfn " Hue: %f Saturation: %f Brightness: %f" h s b

let printAll col colorString =
  printfn "%s" colorString
  printRGB col
  printHSB col

printAll Color.Red "Red"
printAll Color.Black "Black"
printAll Color.White "White"
printAll Color.Gray "Gray"
printAll Color.BlanchedAlmond "BlanchedAlmond"

// Red
//  Red: 255 Green: 0 Blue: 0
//  Hue: 360.000000 Saturation: 1.000000 Brightness: 0.500000
// Black
//  Red: 0 Green: 0 Blue: 0
//  Hue: 0.000000 Saturation: 0.000000 Brightness: 0.000000
// White
//  Red: 255 Green: 255 Blue: 255
//  Hue: 0.000000 Saturation: 0.000000 Brightness: 1.000000
// Gray
//  Red: 128 Green: 128 Blue: 128
//  Hue: 0.000000 Saturation: 0.000000 Brightness: 0.501961
// BlanchedAlmond
//  Red: 255 Green: 235 Blue: 205
//  Hue: 36.000000 Saturation: 1.000000 Brightness: 0.901961
```


### Partial Active Patterns

Active patterns that do not always produce a value are called partial active patterns; they have a return value that is an option type. To define a partial active pattern, you use a wildcard character (_) at the end of the list of patterns inside the banana clips. 


```fsharp 
let (|Integer|_|) (str: string) =
   let mutable intvalue = 0
   if System.Int32.TryParse(str, &intvalue) then Some(intvalue)
   else None

let (|Float|_|) (str: string) =
   let mutable floatvalue = 0.0
   if System.Double.TryParse(str, &floatvalue) then Some(floatvalue)
   else None

let parseNumeric str =
   match str with
     | Integer i -> printfn "%d : Integer" i
     | Float f -> printfn "%f : Floating point" f
     | _ -> printfn "%s : Not matched." str

parseNumeric "1.1"
parseNumeric "0"
parseNumeric "0.0"
parseNumeric "10"
parseNumeric "Something else"

// output
// 1.100000 : Floating point
// 0 : Integer
// 0.000000 : Floating point
// 10 : Integer
// Something else : Not matched.


let err = 1.e-10

let isNearlyIntegral (x:float) = abs (x - round(x)) < err

let (|Square|_|) (x : int) =
  if isNearlyIntegral (sqrt (float x)) then Some(x)
  else None

let (|Cube|_|) (x : int) =
  if isNearlyIntegral ((float x) ** ( 1.0 / 3.0)) then Some(x)
  else None

let findSquareCubes x =
   match x with
       | Cube x & Square _ -> printfn "%d is a cube and a square" x
       | Cube x -> printfn "%d is a cube" x
       | _ -> ()

[ 1 .. 1000 ] |> List.iter (fun elem -> findSquareCubes elem)
// output
// 1 is a cube and a square
// 8 is a cube
// 27 is a cube
// 64 is a cube and a square
// 125 is a cube
// 216 is a cube
// 343 is a cube
// 512 is a cube
// 729 is a cube and a square
// 1000 is a cube

```


### Parameterized Active Patterns

Active patterns always take at least one argument for the item being matched, but they may take additional arguments as well, in which case the name parameterized active pattern applies. Additional arguments allow a general pattern to be specialized.

```fsharp
open System.Text.RegularExpressions

// ParseRegex parses a regular expression and returns a list of the strings that match each group in
// the regular expression.
// List.tail is called to eliminate the first element in the list, which is the full matched expression,
// since only the matches for each group are wanted.
let (|ParseRegex|_|) regex str =
   let m = Regex(regex).Match(str)
   if m.Success
   then Some (List.tail [ for x in m.Groups -> x.Value ])
   else None

// Three different date formats are demonstrated here. The first matches two-
// digit dates and the second matches full dates. This code assumes that if a two-digit
// date is provided, it is an abbreviation, not a year in the first century.
let parseDate str =
   match str with
     | ParseRegex "(\d{1,2})/(\d{1,2})/(\d{1,2})$" [Integer m; Integer d; Integer y]
          -> new System.DateTime(y + 2000, m, d)
     | ParseRegex "(\d{1,2})/(\d{1,2})/(\d{3,4})" [Integer m; Integer d; Integer y]
          -> new System.DateTime(y, m, d)
     | ParseRegex "(\d{1,4})-(\d{1,2})-(\d{1,2})" [Integer y; Integer m; Integer d]
          -> new System.DateTime(y, m, d)
     | _ -> new System.DateTime()

let dt1 = parseDate "12/22/08"
let dt2 = parseDate "1/1/2009"
let dt3 = parseDate "2008-1-15"
let dt4 = parseDate "1995-12-28"

printfn "12/22/08 -> %s" (dt1.ToString()) 
printfn "1/1/2009 -> %s" (dt2.ToString()) 
printfn "2008-1-15 -> %s" (dt3.ToString()) 
printfn "1995-12-28 -> %s" (dt4.ToString())
// output
// 12/22/08 -> 22/12/2008 12:00:00 am
// 1/1/2009 -> 1/1/2009 12:00:00 am
// 2008-1-15 -> 15/1/2008 12:00:00 am
// 1995-12-28 -> 28/12/1995 12:00:00 am
// 2000.12.21 -> 1/1/0001 12:00:00 am
```

Active patterns are not restricted only to pattern matching expressions, you can also use them on let-bindings.

```fsharp
let (|Default|) onNone value =
    match value with
    | None -> onNone
    | Some e -> e

let greet (Default "random citizen" name) =
    printfn "Hello, %s!" name

greet None
greet (Some "George")
// Hello, random citizen!
// Hello, George!
```









<file_sep>+++
date = "2016-12-07T16:56:21+11:00"
title = "React Redux CRM Project"
description = "React-Crm is reusable CRM starter project for real-world business based on React 15.4"
+++


## Summary

**React-Crm** is reusable CRM starter project for real-world business based on React 15.4, React-Redux & Material-UI.


The goal of this starter project is to create reusable project for real-world business. To achieve this target, we need a solution which should include authentication process, restful API feature with token support and simple but elegant UI design. 


## __Features__

* This project is built on the top of React/Redux. 

* The UI part of this project uses Material-UI. 

* This project uses Redux-Thunk to support backend API.

* It uses Json-Server as fake Restful API. (You can simple replace it with your own API)

## Structure of React-Crm

``` ini
path\to\ng4crm
+---config             <-// configuration of dev or prod environment
+---db                 <-// json files for json-server
|   +---db.json        <-// dummy db
|   \---routes.json    <-// configure fake restful api
+---screenshots
+---src                <-// vue components 
|   +---app
|   |   +---_gurad             <-// auth guard for authentication
|   |   +---_models             <-// common models for whole app
|   |   +---_services            <-// common services for whole app
|   |   +---about                <-// about component   
|   |   +---customer              <-// customer component
|   |   +---dashboard            <-// dashboard component  
|   |   +---notfoundpage         <-// notfoundpage component  
|   |   +---login                <-// login component  
|   |   +---order                <-// customer component 
|   |   +---root                <-// root component 
|   |   +---shared                <-// common component for whole app
|   |   +---app.component.ts
|   |   +---app.module.ts
|   |   +---app.routes.ts
|   |   +---app.services.ts
|   |   +---environment.ts
|   |   \---...
|   +---assets         <-//   images  and css from third parties
|   +---styles         <-//   customized css files
|   +---main.browser.aot.ts     
|   +---main.browser.ts  
|   +---polyfills.browser.ts  
|   \---...
\...

```


## Screenshots

> ![Screenshot1](/img/rrcrm-screenshot-1.jpg)

> ![Screenshot2](/img/rrcrm-screenshot-2.jpg)

> ![Screenshot3](/img/rrcrm-screenshot-3.jpg)

> ![Screenshot4](/img/rrcrm-screenshot-4.jpg)


## Browse [Repository](https://github.com/harryho/react-crm.git)



## __Alternatives__

There are two similar projects respectively built on the Vue.js and Angular. If you have interests in those technical stacks. You can find and clone those projects below.

* [Vue2 Crm](/projects/vue2-crm/).
* [Angular4 Crm](/projects/angular4-crm).

<file_sep>+++

title = "Java Note - 7: Stream API"
description="Java Stream"
hidden="true"
+++

## Stream

### Iterations

### Collections and Maps

## Filtering

## Mapping


### LambdaCollectionDemo


### Lambda Collection Map Demo

```java


public class LambdaCollectionMapDemo {

    public static void main(String[] args) {

        // FunctionalInterface 
        System.out.println("x + y:" + engine((x, y) -> x + y));// w w  w .j  av a 2s.  c om
        System.out.println("x - y:" + engine((x, y) -> x * y));
        System.out.println("x * y:" + engine((x, y) -> x / y));
        System.out.println("x / y:" + engine((x, y) -> x % y));

        String[] strArray = new String[] { "abc", "klm", "xyz", "pqr" };
        List list = Arrays.asList(strArray);

        // Default Methods
        list.forEach(System.out::println);

        Arrays.sort(strArray, (first, second) -> first.compareToIgnoreCase(second));
        list = Arrays.asList(strArray);

        System.out.println("After sorting ... ");
        list.forEach(System.out::println);



        // Common Functional Interfaces
        // Runnable
        repeat(5, () -> System.out.println("Hello")) ;

        // UnaryOperator
        UnaryOperator<String> upperCase = str -> str.toUpperCase();
        // BiUnaryOperator
        BinaryOperator<String> concat = (left,right) -> left + right;

        System.out.println( " UnaryOperator upperCase "+upperCase.apply( "hello") );
        System.out.println( "  BinaryOperator<String> concat "+ concat.apply("hello","world"));
        

    }

    private static int engine(Calculator calculator) {
        int x = 2, y = 4;
        return calculator.calculate(x, y);
    }

    public static void repeat(int n, Runnable action) {
        for (int i = 0; i < n; i++)
            action.run();
    }
}

@FunctionalInterface
interface Calculator {
    int calculate(int x, int y);
}

```





<file_sep>+++
title = "AWS: VPC - 2"
description = "VPC -  Route table, Security Group, Network ACL"
weight=4
+++


## VPC Part 2

VPC has an implicit router (implied router), and you use route tables to control where network traffic is directed.

### Route table

* Have up to 200 route tables per VPC
* Have up to 50 route entries per route table
* Each subnet must be associated with only one route table
* The subent (when created) will be associated with main (default) VPC route table
* Can change the subnet association to another route table
* Can NOT delete the main route table
* Every route table in a VPC comes with a default rule that allows all VPC subnets to comminunicate with one another. This rule can NOT be modified or deleted.

### Security Group


* It is a virtual firewall
* It controls traffic at the EC2ss level
* Up to 5 security groups per EC2 
* Stateful, return traffic, of allowed inbound traffic, is allowed, even if there are no rules to allow it. 
* Can only have permit rules, can NOT have deny rules
* Implicit deny rule at the end
* Security group is associated wth EC2's network interface
* Any change on security group takes effect immediately
* Default security groupd can not be deleted
* It is VPC resource, hence, different EC2 in differenet AZs within the same VPC, can have the same security group applied to them.
* It can NOT block a certain range of IP addresses from Internet from gettting to EC2 fleets

#### Default vs Customized Group

The default security group has inbound and outbound rules when created. The inboud rule allows all traffics in from the same security group. The outbound rule allows all traffics to any destination. The customized security group has outbound rule only by default.

### Network ACL

* A default NACL allows all traffic inbound and outbound.
* A cutomized NACL blocks/denies all traffic inbound / outbound by default.
* A NACL can block certain ranges of IP address from a large pool(Internet address for instance)
* A network ACL can associate with multiple subnets. However, a subnet can be associated with ONLY ONE network ACL at a time. When you associate a network ACL with a subnet, the previous association is removed.



### Security Group vs NACL

Security Group | NACL
---------------|----------
Operates at the instance level| Operates at the subnet level
Supports allow rules only| Supports allow rules and deny rules
Is stateful: Return traffic is automatically allowed, regardless of any rules| Is stateless: Return traffic must be explicitly allowed by rules
We evaluate all rules before deciding whether to allow traffic| We process rules in number order when deciding whether to allow traffic
Applies to an instance only if someone specifies the security group when launching the instance, or associates the security group with the instance later on| Automatically applies to all instances in the subnets that it's associated with (therefore, it provides an additional layer of defense if the security group rules are too permissive)


### NAT instance

A network address translation (NAT) instance in a public subnet in your VPC to enable instances in the private subnet to initiate outbound IPv4 traffic to the Internet or other AWS services, but prevent the instances from receiving inbound traffic initiated by someone on the Internet.

Amazon provides Amazon Linux AMIs that are configured to run as NAT instances. These AMIs include the string amzn-ami-vpc-nat in their names, so you can search for them in the Amazon EC2 console.

#### Route table & Security Group setting

* Assumption

    - The CIDR of VPC is 10.0.0.0/16
    - Public subnet is 10.0.1.0/24
    - Private subnet is 10.0.2.0/24
    - Identifier of Nat Instance is __nat-instance-id__

* Custom route table of public subnet

Destination | Target 
---|----
10.0.0.0/16 | local
0.0.0.0/0   | igw-id 


* Main route table

Destination | Target
 ----- | ----
10.0.0.0/16 | local
0.0.0.0/0   | nat-instance-id



### NAT Gateway

A network address translation (NAT) gateway to enable instances in a private subnet to connect to the internet or other AWS services, but prevent the internet from initiating a connection with those instances.

### NAT Instance vs NAT Gateway

Attribute  |  NAT gateway   |  NAT instance
-----------|------------------|--------------
Availability   |    Highly available. NAT gateways in each Availability Zone are implemented with redundancy. Create a NAT gateway in each Availability Zone to ensure zone-independent architecture.   |    Use a script to manage failover between instances.
Bandwidth   |    Can scale up to 45 Gbps.   |    Depends on the bandwidth of the instance type.
Maintenance   |    Managed by AWS. You do not need to perform any maintenance.   |    Managed by you, for example, by installing software updates or operating system patches on the instance.
Performance   |    Software is optimized for handling NAT traffic.   |    A generic Amazon Linux AMI that's configured to perform NAT.
Cost   |    Charged depending on the number of NAT gateways you use, duration of usage, and amount of data that you send through the NAT gateways.   |    Charged depending on the number of NAT instances that you use, duration of usage, and instance type and size.
Type and size   |    Uniform offering; you don’t need to decide on the type or size.   |    Choose a suitable instance type and size, according to your predicted workload.
Public IP addresses   |    Choose the Elastic IP address to associate with a NAT gateway at creation.   |    Use an Elastic IP address or a public IP address with a NAT instance. You can change the public IP address at any time by associating a new Elastic IP address with the instance.
Private IP addresses   |    Automatically selected from the subnet's IP address range when you create the gateway.   |    Assign a specific private IP address from the subnet's IP address range when you launch the instance.
Security groups   |    Cannot be associated with a NAT gateway. You can associate security groups with your resources behind the NAT gateway to control inbound and outbound traffic.   |    Associate with your NAT instance and the resources behind your NAT instance to control inbound and outbound traffic.
Network ACLs   |    Use a network ACL to control the traffic to and from the subnet in which your NAT gateway resides.   |    Use a network ACL to control the traffic to and from the subnet in which your NAT instance resides.
Flow logs   |    Use flow logs to capture the traffic.   |    Use flow logs to capture the traffic.
Port forwarding   |    Not supported.   |    Manually customize the configuration to support port forwarding.
Bastion servers   |    Not supported.   |    Use as a bastion server.
Traffic metrics   |    View CloudWatch metrics for the NAT gateway.   |    View CloudWatch metrics for the instance.
Timeout behavior   |    When a connection times out, a NAT gateway returns an RST packet to any resources behind the NAT gateway that attempt to continue the connection (it does not send a FIN packet).   |    When a connection times out, a NAT instance sends a FIN packet to resources behind the NAT instance to close the connection.
IP fragmentation   | Supports forwarding of IP fragmented packets for the UDP protocol. Does not support fragmentation for the TCP and ICMP protocols. Fragmented packets for these protocols will get dropped. |    Supports reassembly of IP fragmented packets for the UDP, TCP, and ICMP protocols. 


<file_sep>+++
date = "2011-03-09T10:59:31+11:00"
title = "Scrum VS Kanban"
description="Comparison between Scrum  & Kanban"
weight=1
+++

## Scrum VS Kanban

Kanban and scrum are frameworks that help teams adhere to agile principles and get stuff done. Both frameworks will help you build better products(and services) with fewer headaches. While the practices differ, the principles are largely the same.

### Agile 

Agile is a structured and iterative approach to project management and product development. t recognizes the volatility of product development, and provides a methodology for self-organizing teams to respond to change without going off the rails.


### Scrum 

Scrum teams adopt specific roles, create special artifacts, and hold regular ceremonies to keep things moving forward. Their goal is to create learning loops to quickly gather and integrate customer feedback.


### Kanban 

Kanban is all about visualizing your work, limiting work in progress, and maximizing efficiency(or flow). Kanban teams focus on reducing the time it takes to take a project(or user story) from start to finish. They do this by using a kanban board and continuously improving their flow of work.


### Difference 

||Scrum|Kanban
|---| --- |  --- |
Cadence| Regular fixed length sprints (ie, 2 weeks)| Continuous flow
Release methodology| At the end of each sprint| Continuous delivery
Roles| Product owner, scrum master, development team| No required roles
Key metrics| Velocity| Lead time, cycle time, WIP
Change philosophy| Teams should not make changes during the sprint.| Change can happen at any time


<file_sep>Kogan.com Recruitment Quiz
Total Questions: 15

1)
Seven violin students — A, B, C, D, E, F, and G — are to give a recital, and their instructor is deciding the order in which they will perform. Each student will play exactly one piece, a violin solo. In deciding the order of performance, the instructor must observe the following restrictions:

i) E cannot play first or second.

ii) D cannot play until E has played.

iii) Neither A nor F can play seventh.

iv) Either F or G must play immediately after D plays.

v) C must play either immediately after or immediately before B plays.

If C plays first, which one of the following must be true?

- [ ] A plays sixth.
- [ ] E plays third.
- [X] G plays seventh.
- [ ] A plays immediately after F.
- [ ] D plays immediately after E.


2) 
For the above question, if B plays third, what is the latest position in which F can play?

- [ ] First

- [ ] Second

- [ ] Fifth

- [x] Sixth

- [ ] Seventh


3)

Arun leaves his house to go for a run. He starts off by heading 1.5 kilometres north. He then turns to his right and runs a further 500 metres, and then turns to his left. After running a further 1 kilometre, he turns left and runs another 2 kilometres. He finally turns left again and runs for a further 2.5 kilometres.

How far away from his house is he, and in what direction?

- [ ] 1 kilometre west

- [ ] 1.5 kilometres east

- [ ] 1 kilometre east

- [ ] He's back at his house

- [x] 1.5 kilometres west
  
4)

Professor Smith lets his students choose their own lab partners in science class, his only rule is that no two students may work together for more than seven consecutive classes. Alex and Barry have been partners for the past seven consecutive classes. Colin and Doug have worked together for the past three classes. Colin doesn't like Alex and refuses to partner with him.

Who should be assigned to work with Barry?

- [x] Colin

- [ ] Doug

- [ ] <NAME>

- [ ] Alex


1) 
A Work Health & Safety committee needs to appoint a chairperson, which will be determined based on length of tenure in the organisation. Raj joined the company before Jorge, but after Felicity. Harvey has been an employee for longer than Jorge, but not as long as Raj. Samuel was the last employee hired prior to Raj joining the business.  Felicity doesn't want the job, as she is already the chairperson for the social club committee.

Who will be the new Work Health & Safety committee chairperson?


- [ ] Raj

- [ ] Jorge

- [x] Samuel

- [ ] Harvey

- [ ] Felicity


6)
Six school children J, K, L, M, N and O are standing in a circle. J is between N and L. M is to the left of O. K is between O and L.

Who is between J and M?

- [ ] K

- [ ] O

- [ ] L

- [x] N


7)
What day comes two days after the day which comes three days after the day which comes immediately after the day which comes three days after Wednesday?

- [ ] Saturday

- [ ] Wednesday

- [x] Thursday

- [ ] Tuesday

- [ ] Sunday

- [ ] Friday

- [ ] Monday


8)
Amanda loves to eat chocolate but not ice cream while Tim loves to eat potato chips and ice cream. Sally loves to eat potato chips but not the chocolate and Steve loves to eat ice cream but not the potato chips.

If each child loves to eat two of the three snacks, which child has the same preference as Tim?

- [ ] Amanda

- [ ] None of them

- [ ] Steve

- [x] Sally

9) 
Jeremy states: When I went fishing the other day, every fish that I caught was a tuna, and every tuna I saw I caught.

Of the following statements listed below, which one can be concluded from the observations of Jeremy?

 

- [ ] Jeremy did not see any other fish while he was fishing.

- [x] While Jeremy was fishing, he caught no fish other than tuna.

- [ ] In the area that Jeremy fished, there were no other fish.

- [ ] All of the fish that Jeremy saw he caught.

- [ ] Tuna was the only fish that Jeremy saw while he was fishing.


10)
Four defenders in a football match — Defender 1, Defender 2, Defender 3 & Defender 4 — take their positions in this order in a row from right to left. During the match, Defender 1 changes places with Defender 3 and then Defender 3 changes places with Defender 2.

Which defender is now at the left end of the row?

- [ ] Defender 3

- [ ] Defender 2

- [ ] Defender 1

- [x] Defender 4


11)
Identify the two statements necessary to make the following conclusion true:
 __Roger is swimming.__

i) Roger is in the pool during the winter.

ii) Roger is in the pool.

iii) Swimming takes place in the pool during the winter.

iv) All the boys in the pool are swimming.


- [ ] i & ii

- [x] ii & iv

- [ ] iii & iv

- [ ] iv & i


12)  
A mountain climber is slowly making his way up a steep path over difficult terrain. Each day he climbs 3 kilometres, however by the time he wakes up each morning he has somehow slipped backwards by 2 kilometres over the course of the night.

If the path is 30 kilometres long, how many days will it take him to reach the end?

- [ ] 27

- [x] 28

- [ ] 29

- [ ] 30


13)

Four strangers saw someone attacked on the street. Each person gave a different description of the assailant to the police.

Which description is most likely correct?

- [ ] He was short, overweight and middle aged

- [ ] He was tall, overweight and young

- [x] He was short, overweight and young

- [ ] He was short, thin and young

14)  

Six food items are stacked in a cupboard divided into several racks. Mustard sits on the rack right above the salt and the sugar is in the rack right below the apple juice. Apple juice is in the rack right above the mustard and the salt shares the rack with the coffee. Cashews are kept on the lowest rack.



Which item is on the topmost rack?

- [x] Apple juice

- [ ] Coffee

- [ ] Mustard

- [ ] Salt

15) 
    
In the question above, which two other items (aside from salt & coffee) share a rack?

- [ ] Mustard & apple juice

- [ ] Cashews & sugar

- [X] Sugar & mustard
  
- [ ] None of above is correct

<file_sep>+++
title = "Test"
description="Rustlang Introduction: Test"
weight = 5
draft = true
+++


### Test

#### Introduction of test

Tests are Rust functions that verify that the non-test code is functioning in the expected manner. The bodies of test functions typically perform these three actions:

* Set up any needed data or state.
* Run the code you want to test.
* Assert the results are what you expect.


#### Simple test sample

```rs
#[cfg(test)]
mod tests {
    #[test]
    fn it_works() {
        assert_eq!(2 + 2, 4);
    }
}
```

#### The assert Macro

The assert! macro, provided by the standard library, is useful when you want to ensure that some condition in a test evaluates to true. 

However, this is such a common test that the standard library provides a pair of macros—assert_eq! and assert_ne!—to perform conveniently such test, which is to compare the result of the code under test to the value you expect the code to return.

Under the surface, the assert_eq! and assert_ne! macros use the operators == and !=, respectively. When the assertions fail, these macros print their arguments using debug formatting, which means the values being compared must implement the PartialEq and Debug traits. All the primitive types and most of the standard library types implement these traits. For structs and enums that you define, you’ll need to implement PartialEq to assert that values of those types are equal or not equal. You’ll need to implement Debug to print the values when the assertion fails. 


#### Failure Messages

Custom messages are useful to document what an assertion means; when a test fails, you’ll have a better idea of what the problem is with the code.

```rs
#[test]
fn greeting_contains_name() {
    let result = greeting("Carol");
    assert!(
        result.contains("Carol"),
        "Greeting did not contain name, value was `{}`", result
    );
}
```

#### Use should_panic and message

Tests that use should_panic can be imprecise because they only indicate that the code has caused some panic. A should_panic test would pass even if the test panics for a different reason from the one we were expecting to happen. To make should_panic tests more precise, we can add an optional expected parameter to the should_panic attribute. The test harness will make sure that the failure message contains the provided text. 

```rs
impl Guess {
    pub fn new(value: i32) -> Guess {
        if value < 1 {
            panic!("Guess value must be greater than or equal to 1, got {}.",
                   value);
        } else if value > 100 {
            panic!("Guess value must be less than or equal to 100, got {}.",
                   value);
        }

        Guess {
            value
        }
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    #[should_panic(expected = "Guess value must be less than or equal to 100")]
    fn greater_than_100() {
        Guess::new(200);
    }
}
```


#### Using Result<T, E>

Writing tests so they return a Result<T, E> enables you to use the question mark operator in the body of tests, which can be a convenient way to write tests that should fail if any operation within them returns an Err variant.

```rs

#[cfg(test)]
mod tests {
    #[test]
    fn it_works() -> Result<(), String> {
        if 2 + 2 == 4 {
            Ok(())
        } else {
            Err(String::from("two plus two does not equal four"))
        }
    }
}
```


### How to run test













<file_sep>+++

title = "ABC"
description = "ABC - Abstract Base Classes"
weight=9
+++


## ABC - Abstract Base Classes

Abstract base classes complement duck-typing by providing a way to define interfaces when other techniques like hasattr() would be clumsy or subtly wrong (for example with magic methods). ABCs introduce virtual subclasses, which are classes that don’t inherit from a class but are still recognized by isinstance() and issubclass(); see the abc module documentation. Python comes with many built-in ABCs for data structures (in the collections module), numbers (in the numbers module), and streams (in the io module). You can create your own ABCs with the abc module.

## ABCMeta 

Metaclass for defining Abstract Base Classes (ABCs).

Use this metaclass to create an ABC. An ABC can be subclassed directly, and then acts as a mix-in class. You can also register unrelated concrete classes (even built-in classes) and unrelated ABCs as “virtual subclasses” – these and their descendants will be considered subclasses of the registering ABC by the built-in issubclass() function, but the registering ABC won’t show up in their MRO (Method Resolution Order) nor will method implementations defined by the registering ABC be callable (not even via super()).

### register

```python
from abc import ABCMeta

class MyABC(metaclass=ABCMeta):
    pass

MyABC.register(tuple)

assert issubclass(tuple, MyABC)
assert isinstance((), MyABC)
```

### __subclasshook__

Check whether subclass is considered a subclass of this ABC. This means that you can customize the behavior of issubclass further without the need to call register() on every class you want to consider a subclass of the ABC. 

```python

## TODO

```

###  @abc.abstractmethod

A decorator indicating abstract methods.

Using this decorator requires that the class’s metaclass is ABCMeta or is derived from it. A class that has a metaclass derived from ABCMeta cannot be instantiated unless all of its abstract methods and properties are overridden. The abstract methods can be called using any of the normal ‘super’ call mechanisms. abstractmethod() may be used to declare abstract methods for properties and descriptors.

Dynamically adding abstract methods to a class, or attempting to modify the abstraction status of a method or class once it is created, are not supported. The abstractmethod() only affects subclasses derived using regular inheritance; “virtual subclasses” registered with the ABC’s register() method are not affected.


### Sample code


```python

# Source: https://github.com/PythonCharmers/python-future/blob/466bfb2dfa36d865285dc31fe2b0c0a53ff0f181/future/utils/__init__.py#L102-L134

def with_metaclass(meta, *bases):
    """
    Function from jinja2/_compat.py. License: BSD.
    Use it like this::
        class BaseForm(object):
            pass
        class FormType(type):
            pass
        class Form(with_metaclass(FormType, BaseForm)):
            pass
    This requires a bit of explanation: the basic idea is to make a
    dummy metaclass for one level of class instantiation that replaces
    itself with the actual metaclass.  Because of internal type checks
    we also need to make sure that we downgrade the custom metaclass
    for one level to something closer to type (that's why __call__ and
    __init__ comes back from type etc.).
    This has the advantage over six.with_metaclass of not introducing
    dummy classes into the final MRO.
    """

    class Metaclass(meta):
        __call__ = type.__call__
        __init__ = type.__init__

        def __new__(cls, name, this_bases, d):
            if this_bases is None:
                return type.__new__(cls, name, (), d)
            return meta(name, bases, d)

    return Metaclass('temporary_class', None, {})


class Storage(with_metaclass(ABCMeta, object)):
    """
    The abstract base class for all Storages.

    A Storage (de)serializes the current state of the database and stores it in
    some place (memory, file on disk, ...).
    """

    # Using ABCMeta as metaclass allows instantiating only storages that have
    # implemented read and write

    @abstractmethod
    def read(self):
        """
        Read the last stored state.

        Any kind of deserialization should go here.
        Return ``None`` here to indicate that the storage is empty.

        :rtype: dict
        """

        raise NotImplementedError('To be overridden!')

    @abstractmethod
    def write(self, data):
        """
        Write the current state of the database to the storage.

        Any kind of serialization should go here.

        :param data: The current state of the database.
        :type data: dict
        """

        raise NotImplementedError('To be overridden!')

    def close(self):
        """
        Optional: Close open file handles, etc.
        """

        pass


class JSONStorage(Storage):
    """
    Store the data in a JSON file.
    """

    def __init__(self, path, create_dirs=False, **kwargs):
        """
        Create a new instance.

        Also creates the storage file, if it doesn't exist.

        :param path: Where to store the JSON data.
        :type path: str
        """

        super(JSONStorage, self).__init__()
        touch(path, create_dirs=create_dirs)  # Create file if not exists
        self.kwargs = kwargs
        self._handle = open(path, 'r+')

    def close(self):
        self._handle.close()

    def read(self):
        # Get the file size
        self._handle.seek(0, os.SEEK_END)
        size = self._handle.tell()

        if not size:
            # File is empty
            return None
        else:
            self._handle.seek(0)
            return json.load(self._handle)

    def write(self, data):
        self._handle.seek(0)
        serialized = json.dumps(data, **self.kwargs)
        self._handle.write(serialized)
        self._handle.flush()
        self._handle.truncate()


class MemoryStorage(Storage):
    """
    Store the data as JSON in memory.
    """

    def __init__(self,*args, **kwargs):
        """
        Create a new instance.
        """

        super(MemoryStorage, self).__init__()
        self.memory = None

    def read(self):
        return self.memory

    def write(self, data):
        self.memory = data


```





<file_sep>+++
title = "F# C.U.R.S."
description = "F# Class, Unions, Record & Structure"
weight = 4
+++

## Class

Classes are types that represent objects that can have properties, methods, and events.


### Syntax

```
// Class definition:
type [access-modifier] type-name [type-params] [access-modifier] ( parameter-list ) [ as identifier ] =
[ class ]
[ inherit base-type-name(base-constructor-args) ]
[ let-bindings ]
[ do-bindings ]
member-list
...
[ end ]
// Mutually recursive class definitions:
type [access-modifier] type-name1 ...
and [access-modifier] type-name2 ...
...
```

### Constructors

The constructor is code that creates an instance of the class type. In an F# class, there is always a primary constructor whose arguments are described in the parameter-list that follows the type name, and whose body consists of the let (and let rec) bindings at the start of the class declaration and the do bindings that follow. The arguments of the primary constructor are in scope throughout the class declaration.

```fsharp
type MyClass1(x: int, y: int) =
   do printfn "%d %d" x y
   new() = MyClass1(0, 0)
```

### Self Identifier

To define a self identifier for the whole class, use the as keyword after the closing parentheses of the constructor parameter list, and specify the identifier name.

```fsharp
type MyClass2(dataIn) as self =
    let data = dataIn
    do
        self.PrintMessage()
    member this.PrintMessage() =
        printf "Creating MyClass2 with Data %d" data
```

### Generic Type

Generic type parameters are specified in angle brackets (< and >), in the form of a single quotation mark followed by an identifier. Multiple generic type parameters are separated by commas.

### Recursive Type

```fsharp
open System.IO

type Folder(pathIn: string) =
  let path = pathIn
  let filenameArray : string array = Directory.GetFiles(path)
  member this.FileArray = Array.map (fun elem -> new File(elem, this)) filenameArray

and File(filename: string, containingFolder: Folder) =
   member this.Name = filename
   member this.ContainingFolder = containingFolder

let folder1 = new Folder(".")
for file in folder1.FileArray do
   printfn "%s" file.Name
```


## Union

Discriminated unions provide support for values that can be one of a number of named cases, possibly each with different values and types. Discriminated unions are useful for heterogeneous data; data that can have special cases, including valid and error cases; data that varies in type from one instance to another; and as an alternative for small object hierarchies.

### Syntax

```fsharp
[ attributes ]
type [accessibility-modifier] type-name =
    | case-identifier1 [of [ fieldname1 : ] type1 [ * [ fieldname2 : ] type2 ...]
    | case-identifier2 [of [fieldname3 : ]type3 [ * [ fieldname4 : ]type4 ...]

    [ member-list ]
```

### Remarks

```fsharp
type Shape =
    | Rectangle of width : float * length : float
    | Circle of radius : float
    | Prism of width : float * float * height : float
```


### Using Discriminated Unions

```fsharp

type Shape =
  // The value here is the radius.
| Circle of float
  // The value here is the side length.
| EquilateralTriangle of double
  // The value here is the side length.
| Square of double
  // The values here are the height and width.
| Rectangle of double * double


let pi = 3.141592654

let area myShape =
    match myShape with
    | Circle radius -> pi * radius * radius
    | EquilateralTriangle s -> (sqrt 3.0) / 4.0 * s * s
    | Square s -> s * s
    | Rectangle (h, w) -> h * w

let radius = 15.0
let myCircle = Circle(radius)
printfn "Area of circle that has radius %f: %f" radius (area myCircle)

let squareSide = 10.0
let mySquare = Square(squareSide)
printfn "Area of square that has side %f: %f" squareSide (area mySquare)

let height, width = 5.0, 10.0
let myRectangle = Rectangle(height, width)
printfn "Area of rectangle that has height %f and width %f is %f" height width (area myRectangle)

// ---- OUTPUT ----
// Area of circle that has radius 15.000000: 706.858347
// Area of square that has side 10.000000: 100.000000
// Area of rectangle that has height 5.000000 and width 10.000000 is 50.000000
```


### Tree Data Structures

Discriminated unions can be recursive, meaning that the union itself can be included in the type of one or more cases. Recursive discriminated unions can be used to create tree structures.

```fsharp
type Tree =
    | Tip
    | Node of int * Tree * Tree

let rec sumTree tree =
    match tree with
    | Tip -> 0
    | Node(value, left, right) ->
        value + sumTree(left) + sumTree(right)
let myTree = Node(0, Node(1, Node(2, Tip, Tip), Node(3, Tip, Tip)), Node(4, Tip, Tip))
let resultSumTree = sumTree myTree
printfn "%A" resultSumTree
```

{{<mermaid >}}
graph TD;
    A[0] --- B[1]
    A[0] --- C[4]
    B[1] --- D[2]
    B[1] --- E[3]  
{{</mermaid >}}



Discriminated unions work well if the nodes in the tree are heterogeneous.


```fsharp
type Expression =
    | Number of int
    | Add of Expression * Expression
    | Multiply of Expression * Expression
    | Variable of string

let rec Evaluate (env:Map<string,int>) exp =
    match exp with
    | Number n -> n
    | Add (x, y) -> Evaluate env x + Evaluate env y
    | Multiply (x, y) -> Evaluate env x * Evaluate env y
    | Variable id -> env[id]

let environment = Map [ "a", 1; "b", 2; "c", 3 ]

// Create an expression tree that represents
// the expression: a + 2 * b.
let expressionTree1 = Add(Variable "a", Multiply(Number 2, Variable "b"))

// Evaluate the expression a + 2 * b, given the
// table of values for the variables.
let result = Evaluate environment expressionTree1

```

### Members

```fsharp
**open System

type IPrintable =
    abstract Print: unit -> unit

type Shape =
    | Circle of float
    | EquilateralTriangle of float
    | Square of float
    | Rectangle of float * float

    member this.Area =
        match this with
        | Circle r -> Math.PI * (r ** 2.0)
        | EquilateralTriangle s -> s * s * sqrt 3.0 / 4.0
        | Square s -> s * s
        | Rectangle(l, w) -> l * w

    interface IPrintable with
        member this.Print () =
            match this with
            | Circle r -> printfn $"Circle with radius %f{r}"
            | EquilateralTriangle s -> printfn $"Equilateral Triangle of side %f{s}"
            | Square s -> printfn $"Square with side %f{s}"
            | Rectangle(l, w) -> printfn $"Rectangle with length %f{l} and width %f{w}"

```


## Record


Records represent simple aggregates of named values, optionally with members. They can either be structs or reference types. They are reference types by default.

### Syntax

```fsharp
[ attributes ]
type [accessibility-modifier] typename =
    { [ mutable ] label1 : type1;
      [ mutable ] label2 : type2;
      ... }
    [ member-list ]
```


### Remarks

You can use the [<Struct>] attribute to create a struct record rather than a record which is a reference type.

```fsharp
// Labels are separated by semicolons when defined on the same line.
type Point = { X: float; Y: float; Z: float; }

// You can define labels on their own line with or without a semicolon.
type Customer = 
    { First: string
      Last: string;
      SSN: uint32
      AccountNumber: uint32; }

// A struct record.
[<Struct>]
type StructPoint = 
    { X: float
      Y: float
      Z: float }
```


### Pattern Matching

```fsharp
type Point3D = { X: float; Y: float; Z: float }
let evaluatePoint (point: Point3D) =
    match point with
    | { X = 0.0; Y = 0.0; Z = 0.0 } -> printfn "Point is at the origin."
    | { X = xVal; Y = 0.0; Z = 0.0 } -> printfn "Point is on the x-axis. Value is %f." xVal
    | { X = 0.0; Y = yVal; Z = 0.0 } -> printfn "Point is on the y-axis. Value is %f." yVal
    | { X = 0.0; Y = 0.0; Z = zVal } -> printfn "Point is on the z-axis. Value is %f." zVal
    | { X = xVal; Y = yVal; Z = zVal } -> printfn "Point is at (%f, %f, %f)." xVal yVal zVal

evaluatePoint { X = 0.0; Y = 0.0; Z = 0.0 }
evaluatePoint { X = 100.0; Y = 0.0; Z = 0.0 }
evaluatePoint { X = 10.0; Y = 0.0; Z = -1.0 }

// ---- OUTPUT ----
// Point is at the origin.
// Point is on the x-axis. Value is 100.000000.
// Point is at (10.000000, 0.000000, -1.000000).
```


### Members

A common approach is to define a Default static member for easy record construction.
```fsharp 
type Person =
  { Name: string
    Age: int
    Address: string }

    static member Default =
        { Name = "Phillip"
          Age = 12
          Address = "123 happy fun street" }

let defaultPerson = Person.Default

```


If you use a self identifier, that identifier refers to the instance of the record whose member is called:

```fsharp
type Person =
  { Name: string
    Age: int
    Address: string }

    member this.WeirdToString() =
        this.Name + this.Address + string this.Age

let p = { Name = "a"; Age = 12; Address = "abc123" }
let weirdString = p.WeirdToString()
```


### Differences Between Records and Classes

Record fields differ from class fields in that they are automatically exposed as properties, and they are used in the creation and copying of records. Record construction also differs from class construction. In a record type, you cannot define a constructor. Instead, the construction syntax described in this topic applies. Classes have no direct relationship between constructor parameters, fields, and properties.

## Structure

A structure is a compact object type that can be more efficient than a class for types that have a small amount of data and simple behavior.

### Remarks

```fsharp
// In Point3D, three immutable values are defined.
// x, y, and z will be initialized to 0.0.
type Point3D =
    struct
        val x: float
        val y: float
        val z: float
    end

// In Point2D, two immutable values are defined.
// It also has a member which computes a distance between itself and another Point2D.
// Point2D has an explicit constructor.
// You can create zero-initialized instances of Point2D, or you can
// pass in arguments to initialize the values.
type Point2D =
    struct
        val X: float
        val Y: float
        new(x: float, y: float) = { X = x; Y = y }

        member this.GetDistanceFrom(p: Point2D) =
            let dX = (p.X - this.X) ** 2.0
            let dY = (p.Y - this.Y) ** 2.0
            
            dX + dY
            |> sqrt
    end
```


### ByRefLike structs

A "byref-like" struct in F# is a stack-bound value type. It is never allocated on the managed heap. A byref-like struct is useful for high-performance programming, as it is enforced with set of strong checks about lifetime and non-capture. The rules are:

    They can be used as function parameters, method parameters, local variables, method returns.
    They cannot be static or instance members of a class or normal struct.
    They cannot be captured by any closure construct (async methods or lambda expressions).
    They cannot be used as a generic parameter.


```fsharp
open System
open System.Runtime.CompilerServices

[<IsByRefLike; Struct>]
type S(count1: Span<int>, count2: Span<int>) =
    member x.Count1 = count1
    member x.Count2 = count2
```

### ReadOnly structs

You can annotate structs with the IsReadOnlyAttribute attribute.

```fsharp
[<IsReadOnly; Struct>]
type S(count1: int, count2: int) =
    member x.Count1 = count1
    member x.Count2 = count2
```

## When to Use Classes, Unions, Records, and Structures

Given the variety of types to choose from, you need to have a good understanding of what each type is designed for to select the appropriate type for a particular situation. Classes are designed for use in object-oriented programming contexts. Object-oriented programming is the dominant paradigm used in applications that are written for the .NET Framework. If your F# code has to work closely with the .NET Framework or another object-oriented library, and especially if you have to extend from an object-oriented type system such as a UI library, classes are probably appropriate.

If you are not interoperating closely with object-oriented code, or if you are writing code that is self-contained and therefore protected from frequent interaction with object-oriented code, you should consider using a mix of classes, records and discriminated unions. A single, well thought–out discriminated union, together with appropriate pattern matching code, can often be used as a simpler alternative to an object hierarchy.

Records have the advantage of being simpler than classes, but records are not appropriate when the demands of a type exceed what can be accomplished with their simplicity. Records are basically simple aggregates of values, without separate constructors that can perform custom actions, without hidden fields, and without inheritance or interface implementations. Although members such as properties and methods can be added to records to make their behavior more complex, the fields stored in a record are still a simple aggregate of values.

Structures are also useful for small aggregates of data, but they differ from classes and records in that they are .NET value types. Classes and records are .NET reference types. The semantics of value types and reference types are different in that value types are passed by value. This means that they are copied bit for bit when they are passed as a parameter or returned from a function. They are also stored on the stack or, if they are used as a field, embedded inside the parent object instead of stored in their own separate location on the heap. Therefore, structures are appropriate for frequently accessed data when the overhead of accessing the heap is a problem. For more information about structures, see Structs.
<file_sep>+++


date = "2016-08-11T11:59:31+11:00"
title = "iText with C#"
draft = true
+++<file_sep>+++

date = "2016-04-10T14:59:31+11:00"
title = "Database notes, Part-1"
draft = true
+++


## MySql

### Start mysql server
```
mysqld
```

### Start mysql client 
```
mysql -p 3306
```

### stop mysql server
```
mysqladmin -u root -p shutdown
```


### Change mysql root user password

```
mysql --user=root --pass='' 
mysql -e "update user set password=password('') \
where user='root'; flush privileges;"
```

## Back database as SQL Dump 

If it's an entire DB, then:

```bash
mysqldump -u [uname] -p[pass] db_name > db_backup.sql
```

* If it's specific tables within a DB, then:

```bash
$ mysqldump -u [uname] -p[pass] db_name table1 table2 > table_backup.sql
```

## Import database 

```bash
mysql -u username -p -h localhost YOUR-DATABASE < data.sql
```


### Create database YOUR-DATABASE

```sql
-- Drop database if it exists
DROP DATABASE   /*!32312 IF EXISTS*/ YOUR-DATABASE ;
-- Create database with UTF-8 CHARSET 
CREATE DATABASE YOUR-DATABASE CHAR SET utf8 COLLATE 'utf8_unicode_ci';

```

## PostGreSQL

### Start db

```
e:\db\pgsql9\bin\pg_ctl.exe -D "e:\db\pgsql9\data"  -l logFile start
e:\db\pgsql9\bin\pg_ctl.exe -D "e:\db\pgsql9\data" stop
```


## SQL Server

## Collation precedence

```
SELECT * 
FROM TestTab 
WHERE GreekCol = LatinCol COLLATE greek_ci_as;
```
<file_sep>+++
title = "Golang snippets"
description="Golang snippets"
weight=99
+++



### Convenient logging methods 

#### Stringer

```go
package main

import (
    "fmt"
)

// Animal has a Name and an Age to represent an animal.
type Animal struct {
    Name string
    Age  uint
}

// String makes Animal satisfy the Stringer interface.
func (a Animal) String() string {
    return fmt.Sprintf("%v (%d)", a.Name, a.Age)
}

func main() {
    a := Animal{
        Name: "Gopher",
        Age:  2,
    }
    fmt.Println(a) // Gopher (2)
}
```

#### GoStringer


```go
package main

import (
    "fmt"
)

// Address has a City, State and a Country.
type Address struct {
    City    string
    State   string
    Country string
}

// Person has a Name, Age and Address.
type Person struct {
    Name string
    Age  uint
    Addr *Address
}

// GoString makes Person satisfy the GoStringer interface.
// The return value is valid Go code that can be used to reproduce the Person struct.
func (p Person) GoString() string {
    if p.Addr != nil {
        return fmt.Sprintf(
            "Person{Name: %q, Age: %d, Addr: &Address{City: %q, State: %q, Country: %q}}", 
            p.Name, int(p.Age), p.Addr.City, p.Addr.State, p.Addr.Country)
    }
    return fmt.Sprintf("Person{Name: %q, Age: %d}", p.Name, int(p.Age))
}

func main() {
    p1 := Person{
        Name: "Warren",
        Age:  31,
        Addr: &Address{
            City:    "Denver",
            State:   "CO",
            Country: "U.S.A.",
        },
    }
    // If GoString() wasn't implemented, the output of `fmt.Printf("%#v", p1)` would be similar to
    // Person{Name:"Warren", Age:0x1f, Addr:(*main.Address)(0x10448240)}
    fmt.Printf("%#v\n", p1)

    p2 := Person{
        Name: "Theia",
        Age:  4,
    }
    // If GoString() wasn't implemented, the output of `fmt.Printf("%#v", p2)` would be similar to
    // Person{Name:"Theia", Age:0x4, Addr:(*main.Address)(nil)}
    fmt.Printf("%#v\n", p2)

}

// ----- Output ------
// Person{Name: "Warren", Age: 31, Addr: &Address{City: "Denver", State: "CO", Country: "U.S.A."}}
// Person{Name: "Theia", Age: 4}
```


### File

#### Read file 

```go
func main() {
    file, err := os.Open("input.dat")

    if err != nil {
        fmt.Printf("An error occurred on opening the input file\n" +
            "Does the file exist?\n" +
            "Have you got acces to it?\n")
        return
    }
    defer file.Close()
    iReader := bufio.NewReader(file)
    for {
        str, err := iReader.ReadString('\n')
        if err != nil {
            return // error or EOF
        }
        fmt.Printf("The input was: %s", str)
    }
}
```

#### Read & Write with sliced buffer


```go
func cat(f *os.File) {
    const NBUF = 512
    var buf [NBUF]byte
    for {
        switch nr, er := f.Read(buf[:]); true {
        case nr < 0:
            fmt.Fprintf(os.Stderr, "cat: error reading from %s: %s\n", f, er)
            os.Exit(1)
        case nr == 0: // EOF
            return
        case nr > 0:
            if nw, ew := os.Stdout.Write(buf[0:nr]); nw != nr {
                fmt.Fprintf(os.Stderr, "cat: error writing from %s: %s\n", f, ew)
            }
        }
    }
}
```



### TCP Client/Sever

#### Client

```go
func main() {
        arguments := os.Args
        if len(arguments) == 1 {
                fmt.Println("Please provide host:port.")
                return
        }

        CONNECT := arguments[1]
        c, err := net.Dial("tcp", CONNECT)
        if err != nil {
                fmt.Println(err)
                return
        }

        for {
                reader := bufio.NewReader(os.Stdin)
                fmt.Print(">> ")
                text, _ := reader.ReadString('\n')
                fmt.Fprintf(c, text+"\n")

                message, _ := bufio.NewReader(c).ReadString('\n')
                fmt.Print("->: " + message)
                if strings.TrimSpace(string(text)) == "STOP" {
                        fmt.Println("TCP client exiting...")
                        return
                }
        }
}
```

#### Server

```go
func main() {
        arguments := os.Args
        if len(arguments) == 1 {
                fmt.Println("Please provide port number")
                return
        }

        PORT := ":" + arguments[1]
        l, err := net.Listen("tcp", PORT)
        if err != nil {
                fmt.Println(err)
                return
        }
        defer l.Close()

        c, err := l.Accept()
        if err != nil {
                fmt.Println(err)
                return
        }

        for {
                netData, err := bufio.NewReader(c).ReadString('\n')
                if err != nil {
                        fmt.Println(err)
                        return
                }
                if strings.TrimSpace(string(netData)) == "STOP" {
                        fmt.Println("Exiting TCP server!")
                        return
                }

                fmt.Print("-> ", string(netData))
                t := time.Now()
                myTime := t.Format(time.RFC3339) + "\n"
                c.Write([]byte(myTime))
        }
}
```


<file_sep>+++
title = "MySql: SP & Func"
description="MySql Stored Proc  & Function "
+++

## Function - UDF

>  For the UDF mechanism to work, functions must be written in C or C++ and your operating system must support dynamic loading. MySQL source distributions include a file sql/udf_example.cc that defines five UDF functions. Consult this file to see how UDF calling conventions work. The include/mysql_com.h header file defines UDF-related symbols and data structures, although you need not include this header file directly; it is included by mysql.h.

> A UDF contains code that becomes part of the running server, so when you write a UDF, you are bound by any and all constraints that apply to writing server code. For example, you may have problems if you attempt to use functions from the libstdc++ library. T

### Function sample 

* Check the input string is OZ land line 

```sql
DROP FUNCTION IF EXISTS isAusLandLine;
DELIMITER $$
CREATE FUNCTION isAusLandLine (in_value VARCHAR(500)) RETURNS TINYINT
BEGIN
    DECLARE vv_number VARCHAR(500);

    SELECT REPLACE(in_value, '+','') INTO vv_number;

    RETURN 
      CASE 
          WHEN LEFT(vv_number, 2) IN ('02','03','07','08') ss
          THEN LENGTH(vv_number) = 10 ELSE 0 
      END;
END $$

DELIMITER ;

-- SELECT isAusLandLine('03123322') from dual; -- TRUE

-- SELECT isAusLandLine('06123322') from dual; -- FALSE

```

* EXtract the Json to String


```sql
DROP FUNCTION IF EXISTS json_extract_string;

DELIMITER $

CREATE FUNCTION `json_extract_string`(
  p_json text,
  p_key text
) RETURNS varchar(40) CHARSET latin1
  BEGIN
    SET p_json = replace(p_json, '\\"', '"');
    SET p_json = replace(p_json, '" :', '":');
    SET p_json = replace(p_json, ': "', ':"');
    SET p_json = replace(p_json, ': [', ':[');

    SET @pattern_start_type = '"';
    SET @pattern_end_type = '"';
    SET @pattern = CONCAT('"', p_key, '":',@pattern_start_type);

    IF LOCATE(@pattern, p_json) > 0 THEN
      SET @start_i = LOCATE(@pattern, p_json) + CHAR_LENGTH(@pattern);
    ELSE
      SET @pattern_start_type = '[';
      SET @pattern_end_type = ']';
      SET @pattern = CONCAT('"', p_key, '":',@pattern_start_type);
      SET @start_i = LOCATE(@pattern, p_json) + CHAR_LENGTH(@pattern);
    END IF;

    IF @start_i = CHAR_LENGTH(@pattern) THEN
      SET @end_i = 0;
    ELSE
      SET @end_i = LOCATE(@pattern_end_type, p_json, @start_i) - @start_i;
    END IF;
    RETURN SUBSTR(p_json, @start_i, @end_i);
  END $

DELIMITER ;

-- SELECT json_extract_string('{"key": 123}' ) from dual; -- 123

```

## Stord procedure

> MySQL supports stored routines (procedures and functions). A stored routine is a set of SQL statements that can be stored in the server. Once this has been done, clients don't need to keep reissuing the individual statements but can refer to the stored routine instead. 

 Stored routines can be particularly useful in certain situations:

* When multiple client applications are written in different languages or work on different platforms, but need to perform the same database operations.

* When security is paramount. Banks, for example, use stored procedures and functions for all common operations. This provides a consistent and secure environment, and routines can ensure that each operation is properly logged. In such a setup, applications and users would have no access to the database tables directly, but can only execute specific stored routines. 




### Stored proc sample


* Create a stored proc to execute dynamic SQL script based on the previous execution result 

```sql
DROP PROCEDURE IF EXISTS RunIf;

DELIMITER $$

CREATE PROCEDURE RunIf(ifExpr MEDIUMTEXT, execStmt MEDIUMTEXT)
  BEGIN
    SET @sql = concat('select @result := (', ifExpr, ')');
    PREPARE stmt from @sql;
    EXECUTE stmt;
    DEALLOCATE prepare stmt;
    IF (@result = true) THEN
      SET @sql = execStmt;
      PREPARE stmt FROM @sql;
      EXECUTE stmt;
      DEALLOCATE prepare stmt;
    END IF;
  END
$$

DELIMITER ;

-- CALL( CALL RunIf('EXISTS (SELECT * FROM information_schema.columns 
-- WHERE table_schema = DATABASE() AND table_name = \'TargeTable\' 
-- AND column_name = \'description\')',
--  'ALTER TABLE TargeTable DROP COLUMN description');)

```


<file_sep>+++
title = "F# Collections 2"
description = "F# List, Seq & Map"
weight = 5
+++

## List

A list in F# is an ordered, immutable series of elements of the same type. To perform basic operations on lists, use the functions in the List module.

###  Creating & appending
Creating and Initializing Lists


```fsharp
// --- creating ---
let list123 = [ 1; 2; 3 ]

// You can also put line breaks between elements, in which case the semicolons are optional. 
let list123 = [
    1
    2
    3 ]


// Normally, all list elements must be the same type. 
// An exception is that a list in which the elements are specified to be a base type
//  can have elements that are derived types. Thus the following is acceptable, 
// because both Button and CheckBox derive from Control.
let myControlList : Control list = [ new Button(); new CheckBox() ]

// using a range indicated by integers separated by the range operator (..)
let list1 = [ 1 .. 10 ]


// An empty list.
let listEmpty = []

// use a sequence expression to create a list.
let listOfSquares = [ for i in 1 .. 10 -> i*i ]

// ---- appending -------
// You can attach elements to a list by using the :: (cons) operator. 
// If list1 is [2; 3; 4], the following code creates list2 as [100; 2; 3; 4].
let list2 = 100 :: list1

// You can concatenate lists that have compatible types by using the @ operator, as in the following code. 
// If list1 is [2; 3; 4] and list2 is [100; 2; 3; 4], this code creates list3 as [2; 3; 4; 100; 2; 3; 4].
let list3 = list1 @ list2

```

### Properties

The list type supports the following properties:
Property |	Type |	Description
---| ----| ---
Head 	|'T |	The first element.
Empty |	'T list |	A static property that returns an empty list of the appropriate type.
IsEmpty |	bool |	true if the list has no elements.
Item |	'T |	The element at the specified index (zero-based).
Length 	|int 	|The number of elements.
Tail |	'T list |	The list without the first element.

```fsharp 

let list1 = [ 1; 2; 3 ]

// Properties
printfn "list1.IsEmpty is %b" (list1.IsEmpty)
printfn "list1.Length is %d" (list1.Length)
printfn "list1.Head is %d" (list1.Head)
printfn "list1.Tail.Head is %d" (list1.Tail.Head)
printfn "list1.Tail.Tail.Head is %d" (list1.Tail.Tail.Head)
printfn "list1.Item(1) is %d" (list1.Item(1))
```

### Recursion


```fsharp
// simple implmentaton for small list
let rec sum list =
   match list with
   | head :: tail -> head + sum tail
   | [] -> 0

// The previous code works well for small lists, but for larger lists, 
// it could overflow the stack. The following code improves on this 
// code by using an accumulator argument, a standard technique for
// working with recursive functions.
let sum list =
   let rec loop list acc =
       match list with
       | head :: tail -> loop tail (acc + head)
       | [] -> acc
   loop list 0


// The function RemoveAllMultiples is a recursive function that takes two lists. 
// The first list contains the numbers whose multiples will be removed, and 
// the second list is the list from which to remove the numbers.
// Uses this recursive function to eliminate all the non-prime numbers 
// from a list, leaving a list of prime numbers as the result.
let IsPrimeMultipleTest n x =
   x = n || x % n <> 0

let rec RemoveAllMultiples listn listx =
   match listn with
   | head :: tail -> RemoveAllMultiples tail (List.filter (IsPrimeMultipleTest head) listx)
   | [] -> listx


let GetPrimesUpTo n =
    let max = int (sqrt (float n))
    RemoveAllMultiples [ 2 .. max ] [ 1 .. n ]

printfn "Primes Up To %d:\n %A" 100 (GetPrimesUpTo 100)
// output
// Primes Up To 100:
// [2; 3; 5; 7; 11; 13; 17; 19; 23; 29; 31; 37; 41; 43; 47; 53; 59; 61; 67; 71; 73; 79; 83; 89; 97]

```


### Functions

#### Exists

```fsharp

// Use List.exists to determine whether there is an element of a list satisfies a given Boolean expression.
// containsNumber returns true if any of the elements of the supplied list match
// the supplied number.
let containsNumber number list = List.exists (fun elem -> elem = number) list
let list0to3 = [0 .. 3]
printfn "For list %A, contains zero is %b" list0to3 (containsNumber 0 list0to3)
// The output is as follows
// For list [0; 1; 2; 3], contains zero is true

// Use List.exists2 to compare elements in two lists.
// isEqualElement returns true if any elements at the same position in two supplied
// lists match.
let isEqualElement list1 list2 = List.exists2 (fun elem1 elem2 -> elem1 = elem2) list1 list2
let list1to5 = [ 1 .. 5 ]
let list5to1 = [ 5 .. -1 .. 1 ]
if (isEqualElement list1to5 list5to1) then
    printfn "Lists %A and %A have at least one equal element at the same position." list1to5 list5to1
else
    printfn "Lists %A and %A do not have an equal element at the same position." list1to5 list5to1

// The output is as follows
// Lists [1; 2; 3; 4; 5] and [5; 4; 3; 2; 1] have at least one equal element at the same position.

// You can use List.forall if you want to test whether all the elements of a list meet a condition.
let isAllZeroes list = List.forall (fun elem -> elem = 0.0) list
printfn "%b" (isAllZeroes [0.0; 0.0])
printfn "%b" (isAllZeroes [0.0; 1.0])

// The output is as follows
// true
// false

// Similarly, List.forall2 determines whether all elements in the corresponding 
// positions in two lists satisfy a Boolean expression that involves each pair of elements.


let listEqual list1 list2 = List.forall2 (fun elem1 elem2 -> elem1 = elem2) list1 list2
printfn "%b" (listEqual [0; 1; 2] [0; 1; 2])
printfn "%b" (listEqual [0; 0; 0] [0; 1; 0])

// The output is as follows
// true
// false

```


#### Sort

```fsharp
// use of List.sort.
let sortedList1 = List.sort [1; 4; 8; -2; 5]
printfn "%A" sortedList1

// The output is as follows
// [-2; 1; 4; 5; 8]

// use of List.sortBy.
let sortedList2 = List.sortBy (fun elem -> abs elem) [1; 4; 8; -2; 5]
printfn "%A" sortedList2

// The output is as follows:
// [1; -2; 4; 5; 8]

// The next example demonstrates the use of List.sortWith. In this example, the custom comparison function compareWidgets is used to first compare one field of a custom type, and then another when the values of the first field are equal.
F#

type Widget = { ID: int; Rev: int }

let compareWidgets widget1 widget2 =
   if widget1.ID < widget2.ID then -1 else
   if widget1.ID > widget2.ID then 1 else
   if widget1.Rev < widget2.Rev then -1 else
   if widget1.Rev > widget2.Rev then 1 else
   0

let listToCompare = [
    { ID = 92; Rev = 1 }
    { ID = 110; Rev = 1 }
    { ID = 100; Rev = 5 }
    { ID = 100; Rev = 2 }
    { ID = 92; Rev = 1 }
    ]

let sortedWidgetList = List.sortWith compareWidgets listToCompare
printfn "%A" sortedWidgetList

// The output is as follows:
// [{ID = 92;
// Rev = 1;}; {ID = 92;
// Rev = 1;}; {ID = 100;
// Rev = 2;}; {ID = 100;
// Rev = 5;}; {ID = 110;
// Rev = 1;}]
```

#### Search

```fsharp
// The simplest, List.find, enables you to find the first element that matches a given condition.
let isDivisibleBy number elem = elem % number = 0
let result = List.find (isDivisibleBy 5) [ 1 .. 100 ]
printfn "%d " result
// output
// 5

// If the elements must be transformed first, call List.pick, which takes 
// a function that returns an option, and looks for the first option 
// value that is Some(x). Instead of returning the element, List.pick 
// returns the result x. If no matching element is found, List.pick throws 
// System.Collections.Generic.KeyNotFoundException.
let valuesList = [ ("a", 1); ("b", 2); ("c", 3) ]

let resultPick = List.pick (fun elem ->
                    match elem with
                    | (value, 2) -> Some value
                    | _ -> None) valuesList
printfn "%A" resultPick

The output is as follows:
Console

"b"

// Another group of search operations, List.tryFind and related functions, 
// return an option value. The List.tryFind function returns the first element
// of a list that satisfies a condition if such an element exists, but 
// the option value None if not. The variation List.tryFindIndex returns
// the index of the element, if one is found, rather than the element 
// itself. These functions are illustrated in the following code.


let list1d = [1; 3; 7; 9; 11; 13; 15; 19; 22; 29; 36]
let isEven x = x % 2 = 0
match List.tryFind isEven list1d with
| Some value -> printfn "The first even value is %d." value
| None -> printfn "There is no even value in the list."

match List.tryFindIndex isEven list1d with
| Some value -> printfn "The first even value is at position %d." value
| None -> printfn "There is no even value in the list."
// The output is as follows:
// The first even value is 22.
// The first even value is at position 8.

```

### List & Tuple

Lists that contain tuples can be manipulated by zip and unzip functions. These functions combine two lists of single values into one list of tuples or separate one list of tuples into two lists of single values. 

```fsharp 
// ------------------ zip ---------------------
// The simplest List.zip function takes two lists of single elements and 
// produces a single list of tuple pairs. 
let list1 = [ 1; 2; 3 ]
let list2 = [ -1; -2; -3 ]
let listZip = List.zip list1 list2
printfn "%A" listZip
// The output is as follows
// [(1, -1); (2, -2); (3; -3)]


// Another version, List.zip3, takes three lists of single elements and produces a 
// single list of tuples that have three elements. 
let list3 = [ 0; 0; 0]
let listZip3 = List.zip3 list1 list2 list3
printfn "%A" listZip3
// The output is as follows
// [(1, -1, 0); (2, -2, 0); (3, -3, 0)]


// ------------------ unzip ---------------------

//  use of List.unzip.
let lists = List.unzip [(1,2); (3,4)]
printfn "%A" lists
printfn "%A %A" (fst lists) (snd lists)
// The output is as follows
// ([1; 3], [2; 4])
// [1; 3] [2; 4]


// use of List.unzip3.
let listsUnzip3 = List.unzip3 [(1,2,3); (4,5,6)]
printfn "%A" listsUnzip3
// The output is as follows
// ([1; 4], [2; 5], [3; 6])


```

### Operating

```fsharp
// The simplest is List.iter, which enables you to call a function on every 
// element of a list. Variations include List.iter2, which enables you to 
// perform an operation on elements of two lists, List.iteri, which is like 
// List.iter except that the index of each element is passed as an argument 
// to the function that is called for each element, and List.iteri2, which 
// is a combination of the functionality of List.iter2 and List.iteri.  
let list1 = [1; 2; 3]
let list2 = [4; 5; 6]
List.iter (fun x -> printfn "List.iter: element is %d" x) list1
List.iteri(fun i x -> printfn "List.iteri: element %d is %d" i x) list1
List.iter2 (fun x y -> printfn "List.iter2: elements are %d %d" x y) list1 list2
List.iteri2 (fun i x y ->
                printfn "List.iteri2: element %d of list1 is %d element %d of list2 is %d"
                  i x i y)
            list1 list2

// The output is as follows:
// List.iter: element is 1
// List.iter: element is 2
// List.iter: element is 3
// List.iteri: element 0 is 1
// List.iteri: element 1 is 2
// List.iteri: element 2 is 3
// List.iter2: elements are 1 4
// List.iter2: elements are 2 5
// List.iter2: elements are 3 6
// List.iteri2: element 0 of list1 is 1; element 0 of list2 is 4
// List.iteri2: element 1 of list1 is 2; element 1 of list2 is 5
// List.iteri2: element 2 of list1 is 3; element 2 of list2 is 6


// Another frequently used function that transforms list elements is List.map, 
// which enables you to apply a function to each element of a list and put all 
// the results into a new list. 

// List.map2 and List.map3 are variations that take multiple lists. 
// The only difference between List.mapi2 and List.mapi is that List.mapi2 
// works with two lists. The following example illustrates List.map.

let list1 = [1; 2; 3]
let newList = List.map (fun x -> x + 1) list1
printfn "%A" newList
// The output is as follows:
// [2; 3; 4]


// use of List.map2.
let list1 = [1; 2; 3]
let list2 = [4; 5; 6]
let sumList = List.map2 (fun x y -> x + y) list1 list2
printfn "%A" sumList
// The output is as follows
// [5; 7; 9]


// use of List.map3.
let newList2 = List.map3 (fun x y z -> x + y + z) list1 list2 [2; 3; 4]
printfn "%A" newList2
// The output is as follows:
// [7; 10; 13]


// use of List.mapi.
let newListAddIndex = List.mapi (fun i x -> x + i) list1
printfn "%A" newListAddIndex
// The output is as follows
// [1; 3; 5]


// use of List.mapi2.
let listAddTimesIndex = List.mapi2 (fun i x y -> (x + y) * i) list1 list2
printfn "%A" listAddTimesIndex
// The output is as follows
// [0; 7; 18]



// List.collect is like List.map, except that each element produces a list 
// and all these lists are concatenated into a final list. In the following 
// code, each element of the list generates three numbers.
let collectList = List.collect (fun x -> [for i in 1..3 -> x * i]) list1
printfn "%A" collectList

// The output is as follows:
// [1; 2; 3; 2; 4; 6; 3; 6; 9]

// use List.filter, which takes a Boolean condition and produces a new list 
// that consists only of elements that satisfy the given condition.
let evenOnlyList = List.filter (fun x -> x % 2 = 0) [1; 2; 3; 4; 5; 6]

// The resulting list is [2; 4; 6].

// A combination of map and filter, List.choose enables you to transform 
// and select elements at the same time. List.choose applies a function that 
// returns an option to each element of a list, and returns a new list of 
// the results for elements when the function returns the option value Some.

// use of List.choose to select capitalized words out of a list of words.
let listWords = [ "and"; "Rome"; "Bob"; "apple"; "zebra" ]
let isCapitalized (string1:string) = System.Char.IsUpper string1[0]
let results = List.choose (fun elem ->
    match elem with
    | elem when isCapitalized elem -> Some(elem + "'s")
    | _ -> None) listWords
printfn "%A" results
// The output is as follows:
// ["Rome's"; "Bob's"]

```

### Fold & Scan

The fold and scan operations are like List.iter and List.map in that you invoke a function on each element, but these operations provide an additional parameter called the accumulator that carries information through the computation.

```fsharp
let sumList list = List.fold (fun acc elem -> acc + elem) 0 list
printfn "Sum of the elements of list %A is %d." [ 1 .. 3 ] (sumList [ 1 .. 3 ])
// output
// Sum of the elements of list [1; 2; 3] is 6.

// The following example computes the average of a list.
let averageList list = (List.fold (fun acc elem -> acc + float elem) 0.0 list / float list.Length)

// The following example computes the standard deviation of a list.
// The standard deviation is computed by taking the square root of the
// sum of the variances, which are the differences between each value
// and the average.
let stdDevList list =
    let avg = averageList list
    sqrt (List.fold (fun acc elem -> acc + (float elem - avg) ** 2.0 ) 0.0 list / float list.Length)

let testList listTest =
    printfn "List %A average: %f stddev: %f" listTest (averageList listTest) (stdDevList listTest)

testList [1; 1; 1]
testList [1; 2; 1]
testList [1; 2; 3]
// output
// List [1; 1; 1] average: 1.000000 stddev: 0.000000
// List [1; 2; 1] average: 1.333333 stddev: 0.471405
// List [1; 2; 3] average: 2.000000 stddev: 0.816497



// List.fold is the same as to List.iter when the accumulator is not used.
let printList list = List.fold (fun acc elem -> printfn "%A" elem) () list
printList [0.0; 1.0; 2.5; 5.1 ]
// output
// 0.0
// 1.0
// 2.5
// 5.1



// The following example uses List.fold to reverse a list.
// The accumulator starts out as the empty list, and the function uses the cons operator
// to add each successive element to the head of the accumulator list, resulting in a
// reversed form of the list.
let reverseList list = List.fold (fun acc elem -> elem::acc) [] list
printfn "%A" (reverseList [1 .. 10])
// output
// [10; 9; 8; 7; 6; 5; 4; 3; 2; 1]

// Use List.fold2 to perform computations over two lists (of equal size) at the same time.
// Example: Sum the greater element at each list position.
let sumGreatest list1 list2 = List.fold2 (fun acc elem1 elem2 ->
                                              acc + max elem1 elem2) 0 list1 list2

let sum = sumGreatest [1; 2; 3] [3; 2; 1]
printfn "The sum of the greater of each pair of elements in the two lists is %d." sum
// output
// The sum of the greater of each pair of elements in the two lists is 8.



// List.fold and List.scan differ in that List.fold returns the final value of 
// the extra parameter, but List.scan returns the list of the intermediate 
// values (along with the final value) of the extra parameter.


// Discriminated union type that encodes the transaction type.
type Transaction =
    | Deposit
    | Withdrawal

let transactionTypes = [Deposit; Deposit; Withdrawal]
let transactionAmounts = [100.00; 1000.00; 95.00 ]
let initialBalance = 200.00

// Use fold2 to perform a calculation on the list to update the account balance.
let endingBalance = List.fold2 (fun acc elem1 elem2 ->
                                match elem1 with
                                | Deposit -> acc + elem2
                                | Withdrawal -> acc - elem2)
                                initialBalance
                                transactionTypes
                                transactionAmounts
printfn "%f" endingBalance
// output 
// 1205.000000

// For a calculation like summation, List.fold and List.foldBack have the same effect because the result does not depend on the order of traversal. In the following example, List.foldBack is used to add the elements in a list.
let sumListBack list = List.foldBack (fun elem acc -> acc + elem) list 0
printfn "%d" (sumListBack [1; 2; 3])
// output: 6


// For a calculation in which the order of traversal is important, fold and foldBack have different
// results. For example, replacing fold with foldBack in the listReverse function
// produces a function that copies the list, rather than reversing it.
let copyList list = List.foldBack (fun elem acc -> elem::acc) list []
printfn "%A" (copyList [1 .. 10])
// output: [1; 2; 3; 4; 5; 6; 7; 8; 9; 10]


// The following example returns to the bank account example. This time a new 
// transaction type is added: an interest calculation. The ending balance now 
// depends on the order of transactions.
type Transaction2 =
    | Deposit
    | Withdrawal
    | Interest

let transactionTypes2 = [Deposit; Deposit; Withdrawal; Interest]
let transactionAmounts2 = [100.00; 1000.00; 95.00; 0.05 / 12.0 ]
let initialBalance2 = 200.00

// Because fold2 processes the lists by starting at the head element,
// the interest is calculated last, on the balance of 1205.00.
let endingBalance2 = List.fold2 (fun acc elem1 elem2 ->
                                match elem1 with
                                | Deposit -> acc + elem2
                                | Withdrawal -> acc - elem2
                                | Interest -> acc * (1.0 + elem2))
                                initialBalance2
                                transactionTypes2
                                transactionAmounts2
printfn "%f" endingBalance2
// output
// 1210.020833

// Because foldBack2 processes the lists by starting at end of the list,
// the interest is calculated first, on the balance of only 200.00.
let endingBalance3 = List.foldBack2 (fun elem1 elem2 acc ->
                                match elem1 with
                                | Deposit -> acc + elem2
                                | Withdrawal -> acc - elem2
                                | Interest -> acc * (1.0 + elem2))
                                transactionTypes2
                                transactionAmounts2
                                initialBalance2
printfn "%f" endingBalance3
// output
// 1205.833333
```


<file_sep>+++

title = "Collection"
description = "Collection"
weight=6
+++

## Collection

### Collection protocols

* To implement a protocol, objects must support certain operations.
* Most collections implement container, sized and iterable.
* All except dict and set are sequences
* __Container__: membership testing using `in` and `not in`
* __Sized__: Determine number of elements with `len(s)`
* __Iterable__: Can produce an iterator with `iter(s)`, e.g. 

            for item in iterable:
                do_something(item) `len(s)`

* __Sequence__: 

    * Retrieve elements by index, e.g. `item = seq[index]`
    * Find items by value `index = seq.index(item)`
    * Count items `num = seq.count(item)`
    * Produce a reversed sequence `r = reversed(seq)`

* __Set__: set algebra operations, including method and infix operators . e.g.

    * subset
    * proper subset
    * equal
    * not equal
    * proper superset
    * superset
    * intersections
    * union
    * symmetric difference
    * difference

* Built-in collections


Protocol | Implementing Collections
---------|----
Container |str, list, range, tuple, set, bytes, dict
Sized | str, list, range, tuple, set, bytes, dict
Iterable | str, list, range, tuple, set, bytes, dict
Sequence | str, list, range, tuple, bytes
Set | set
Mutable Sequence | list
Mutable Set | set
Mutable Mapping | dict

### Example

* sortedset 


```python
from bisect import bisect_left
from collections.abc import Sequence, Set
from itertools import chain


class SortedSet(Sequence, Set):

    def __init__(self, items=None):
        self._items = sorted(set(items)) if items is not None else []

    def __contains__(self, item):
        try:
            self.index(item)
            return True
        except ValueError:
            return False

    def __len__(self):
        return len(self._items)

    def __iter__(self):
        return iter(self._items)

    def __getitem__(self, index):
        result = self._items[index]
        return SortedSet(result) if isinstance(index, slice) else result

    def __repr__(self):
        return "SortedSet({})".format(repr(self._items) if self._items else '')

    def __eq__(self, rhs):
        if not isinstance(rhs, SortedSet):
            return False
        return self._items == rhs._items

    def _is_unique_and_sorted(self):
        return all(self[i] < self[i + 1] for i in range(len(self) - 1))

    def index(self, item):
        assert self._is_unique_and_sorted()
        index = bisect_left(self._items, item)
        if (index != len(self._items)) and self._items[index] == item:
            return index
        raise ValueError("{} not found".format(repr(item)))

    def count(self, item):
        assert self._is_unique_and_sorted()
        return int(item in self._items)

    def __add__(self, rhs):
        return SortedSet(chain(self._items, rhs._items))

    def __mul__(self, rhs):
        return SortedSet(self) if rhs > 0 else SortedSet()

    def __rmul__(self, lhs):
        return self * lhs

    def issubset(self, iterable):
        return self <= SortedSet(iterable)

    def issuperset(self, iterable):
        return self >= SortedSet(iterable)

    def intersection(self, iterable):
        return self & SortedSet(iterable)

    def union(self, iterable):
        return self | SortedSet(iterable)

    def symmetric_difference(self, iterable):
        return self ^ SortedSet(iterable)

    def difference(self, iterable):
        return self - SortedSet(iterable)
```


* test case


```python
import unittest
from sorted_set import SortedSet


class TestConstruction(unittest.TestCase):

    def test_empty(self):
        s = SortedSet([])

    def test_from_sequence(self):
        s = SortedSet([7, 8, 3, 1])

    def test_with_duplicates(self):
        s = SortedSet([8, 8, 8])

    def test_from_iterable(self):
        def gen6842():
            yield 6
            yield 8
            yield 4
            yield 2
        g = gen6842()
        s = SortedSet(g)

    def test_default_empty(self):
        s = SortedSet()


class TestContainerProtocol(unittest.TestCase):

    def setUp(self):
        self.s = SortedSet([6, 7, 3, 9])

    def test_positive_contained(self):
        self.assertTrue(6 in self.s)

    def test_negative_contained(self):
        self.assertFalse(2 in self.s)

    def test_positive_not_contained(self):
        self.assertTrue(5 not in self.s)

    def test_negative_not_contained(self):
        self.assertFalse(9 not in self.s)


class TestSizedProtocol(unittest.TestCase):

    def test_empty(self):
        s = SortedSet()
        self.assertEqual(len(s), 0)

    def test_one(self):
        s = SortedSet([42])
        self.assertEqual(len(s), 1)

    def test_ten(self):
        s = SortedSet(range(10))
        self.assertEqual(len(s), 10)

    def test_with_duplicates(self):
        s = SortedSet([5, 5, 5])
        self.assertEqual(len(s), 1)


class TestIterableProtocol(unittest.TestCase):

    def setUp(self):
        self.s = SortedSet([7, 2, 1, 1, 9])

    def test_iter(self):
        i = iter(self.s)
        self.assertEqual(next(i), 1)
        self.assertEqual(next(i), 2)
        self.assertEqual(next(i), 7)
        self.assertEqual(next(i), 9)
        self.assertRaises(StopIteration, lambda: next(i))

    def test_for_loop(self):
        index = 0
        expected = [1, 2, 7, 9]
        for item in self.s:
            self.assertEqual(item, expected[index])
            index += 1


class TestSequenceProtocol(unittest.TestCase):

    def setUp(self):
        self.s = SortedSet([1, 4, 9, 13, 15])

    def test_index_zero(self):
        self.assertEqual(self.s[0], 1)

    def test_index_four(self):
        self.assertEqual(self.s[4], 15)

    def test_index_one_beyond_the_end(self):
        with self.assertRaises(IndexError):
            self.s[5]

    def test_index_minus_one(self):
        self.assertEqual(self.s[-1], 15)

    def test_index_minus_five(self):
        self.assertEqual(self.s[-5], 1)

    def test_index_one_before_the_beginning(self):
        with self.assertRaises(IndexError):
            self.s[-6]

    def test_slice_from_start(self):
        self.assertEqual(self.s[:3], SortedSet([1, 4, 9]))

    def test_slice_to_end(self):
        self.assertEqual(self.s[3:], SortedSet([13, 15]))

    def test_slice_empty(self):
        self.assertEqual(self.s[10:], SortedSet())

    def test_slice_arbitrary(self):
        self.assertEqual(self.s[2:4], SortedSet([9, 13]))

    def test_slice_full(self):
        self.assertEqual(self.s[:], self.s)


class TestReprProtocol(unittest.TestCase):

    def test_repr_empty(self):
        s = SortedSet()
        self.assertEqual(repr(s), "SortedSet()")

    def test_repr_some(self):
        s = SortedSet([42, 40, 19])
        self.assertEqual(repr(s), "SortedSet([19, 40, 42])")


if __name__ == '__main__':
    unittest.main()
```



<file_sep>+++
title = "Azure: CLI - 1"
weight = 4
description="Introduction of Azure CLI"
+++


## Azure Cli 

The Azure Command-Line Interface (CLI) is a cross-platform command-line tool to connect to Azure and execute administrative commands on Azure resources. It allows the execution of commands through a terminal using interactive command-line prompts or a script.


### Install Azure CLI on macOS

* Mac with Intel CPU 

```sh
brew update && brew install azure-cli
```


### Install Azure Cli with Docker

* Use `docker run`

```sh
docker run -it mcr.microsoft.com/azure-cli
```

* Run with SSH key

```sh
docker run -it -v ${HOME}/.ssh:/root/.ssh mcr.microsoft.com/azure-cli
```




### Get Started


* Sign In

```
az login
```

* Show accounts

```
az account list 
az account show

```

* Setup default subscription

```
az account set --sbuscription XXXX-XXXXX-xXXXX-XXX
```

* Show App Services

```
az appservice plan list --query-examples
```


* Show Web Apps

```
az webapp  list --query-examples
```


<file_sep>+++
date = "2016-04-10T14:59:31+11:00"
title = "Scrapy Notes"
draft = true
+++


$ pip install scrapy
$ cat > myspider.py <<EOF
import scrapy

class BlogSpider(scrapy.Spider):
    name = 'blogspider'
    start_urls = ['https://blog.scrapinghub.com']

    def parse(self, response):
        for title in response.css('h2.entry-title'):
            yield {'title': title.css('a ::text').extract_first()}

        next_page = response.css('div.prev-post > a ::attr(href)').extract_first()
        if next_page:
            yield scrapy.Request(response.urljoin(next_page), callback=self.parse)
EOF
$ scrapy runspider myspider.py


Command "c:\apps\python3\python3.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\
\Harry\\AppData\\Local\\Temp\\pip-build-juixotv0\\Twisted\\setup.py';f=getattr(tokenize, 'op
en', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file
__, 'exec'))" install --record C:\Users\Harry\AppData\Local\Temp\pip-w4_uq4jb-record\install
-record.txt --single-version-externally-managed --compile" failed with error code 1 in C:\Us
ers\Harry\AppData\Local\Temp\pip-build-juixotv0\Twisted\
<file_sep>+++
title = "Cron Job Note - 1"
description = "Common Cron Job examples"
+++

## Introduction

Cron job is one of most common techniques used on every Unix / Linux. 

## Common Use Cases

### House keeping - Clean up the backup

* Run every night to remove the daily backup tar ball
* Assumption:
    
    * The script file named  `housekeeping.sh`.
    * The script sits inside folder `bin` which is under your user account.
    * All backup files have the .tar as extension.
    * The script will check the backup tar balls within the folder `bacup`, which is under your user account as well.
    * The script will remove the latest backup if total backup files are over 5.

* Cron Job setting - 

```bash
# Replace the user_id with you actual user name
20 15 * * * <user_id> /home/<user_id>/bin/housekeeping 2>&1  | logger
```

* The script file


```bash
#!/bin/bash

# crontab config 

# 20 15 * * * /home/${USER}/bin/housekeeping 2>&1 

# script location: /home/$(USER}/bin
# log file location: /home/$(USER}/log

logfile=/home/${USER}/log/housekeeping.log;
backup_dir=/home/${USER}/data/backups

# Log the start time
echo $(date)>> $logfile;

fln=$(($(find ${backup_dir} -type f -name "*.tar" | egrep -i ".tar" | wc -l)+0));

# Log the number of files 
echo $fln>>$logfile;

if [[ $fln -ge 6 ]]; then
  for t in $(find ${backup_dir} -type f -name "*.tar" | sort | egrep -m1 ".tar");
  do
    rm -rf   $t;
    # Log the removed file
    echo "remove $t" >> $logfile;
  done;
fi;

```


### Backup the database

#### Backup the MySql database nightly

* The database backup dump will be stored into the folder named after the brand name, e.g. the folder facebook for Facebook

* The script of backup  `backup.sh`

    ```bash

    # Every Backup directory has file named ENVIRONMENT
    # 
    # ENVIRONMENT contains the setting like db, server, etc.
    #
    #    DB_HOST=your_database_host_server
    #    DATABASE=the_target_database
    #    DB_USER=db_login_id
    #    DB_PASSWORD=<PASSWORD>
    #    BACKUPS_TO_KEEP=5
    #


    # THIS SCRIPT MAY ONLY BE RUN ONCE EACH DAY
    # Usge:
    #   ./backup.sh facebook 
    #
    [[ -z "$1" ]] && echo "Must run script with brand name, which has matching folder in /var/backup/" && exit 1

    BACKUP_DIR=/var/backup/${1}
    source ${BACKUP_DIR}/ENVIRONMENT

    LATEST_DUMP_TARBALL=latest-dump.tgz
    DATE=$(date +%d-%m-%Y) # dd-mm-yyyy
    LATEST_BACKUP_DIR=${BACKUP_DIR}/${DATE}

    echo "$(date): Beginning backup of ${DATABASE} on ${DB_HOST}"
    mkdir -p ${LATEST_BACKUP_DIR}
    cd ${LATEST_BACKUP_DIR}
    START=$(date +%s)

    # Set procedure definer to ussadmin
    SQL="UPDATE mysql.proc p \
    SET definer = '${DB_USER}@%' \
    WHERE db = '${DATABASE}' \
    AND definer <> '${DB_USER}@%';"

    mysql -h ${DB_HOST} -P 3306 ${DATABASE} -u ${DB_USER} -p${DB_PASSWORD} -e"${SQL}"

    mysqldump -h ${DB_HOST} -P 3306 ${DATABASE} \
    --routines \
    --lock-tables=false \
    -u ${DB_USER} -p${DB_PASSWORD} > ./dump.sql

    EXPORT_RESULT=$?

    END=$(date +%s)
    echo "Export duration: $(( $END - $START ))s"

    echo "Gzipping dump file into tarball"

    START=$(date +%s)
    tar czf dump.tgz dump.sql
    [[ $? ]] &&  rm -f dump.sql
    END=$(date +%s)

    echo "Gzipping duration: $(( $END - $START ))s"

    echo "Creating ${BACKUP_DIR}/${LATEST_DUMP_TARBALL}"

    if [[ ${EXPORT_RESULT} ]]; then

        if [[ -f ${BACKUP_DIR}/${LATEST_DUMP_TARBALL} ]]; then
            rm --force ${BACKUP_DIR}/${LATEST_DUMP_TARBALL};
        fi

        ln -s ${LATEST_BACKUP_DIR}/dump.tgz ${BACKUP_DIR}/${LATEST_DUMP_TARBALL};

    else
        echo "Export operation did not return 0, skipping symlink update"
    fi

    echo "$(date): Completed backup for ${DATABASE} on ${DB_HOST}"


    ```
<file_sep>+++

date = "2019-12-07T16:56:21+11:00"
title = " Docker Kits"

description = "The repository is the collection of kits built on the top of docker image. "
+++

## Docker Kits

The repository is the collection of kits built on the top of docker image. At the very begiinning, the repository was built for kits used for infrastructure. It is about to expand the scope from infrastructure to simplify the integration test, even build for speical development.

Repository name in Docker Hub: [harryh00/docker-kits](https://hub.docker.com/r/harryh00/docker-kits)


## Why chooses docker

Docker is a tool designed to make it easier to create, deploy, and run applications by using containers. Containers allow a developer to package up an application with all of the parts it needs, such as libraries and other dependencies, and deploy it as one package. 

## Benefits

Containers work a little like VMs, but in a far more specific and granular way. The benefits of Docker containers show up in many places. Here are some of the advantages of Docker used as tool kits:

* Docker enables more efficient use of system resources
* Docker enables faster software delivery cycles
* Docker enables application portability

## Images and tags

### Alpine base kit

* harryh00/docker-kits:alpine-ansible
* harryh00/docker-kits:alpine-terraform
* harryh00/docker-kits:alpine-k8s
* harryh00/docker-kits:alpine-aws2


### Ubuntu base kit

* harryh00/docker-kits:ubuntu-ansible



### CentOS base kit

* harryh00/docker-kits:centos-ansible



## How to use

* Sample 1: use ansible kit

```
# one-off use
docker run --rm -it harryh00/docker-kits:alpine-ansible  ansible --version 

# use the kit  without install ansible
docker run --rm -it  -v ${PWD}:/app   -w /app harryh00/docker-kits:alpine-ansible   /bin/bash 

```

* Sample 2: use aws cli kit 

```
# mount the local aws config to aws cli kit 
docker run --rm -it -v ~/.aws:/root/.aws  harryh00/docker-kits:alpine-aws2
```

* Sample 3: use k8s cli kit 

```
docker run --rm -it -v ~/.kube:/root/.kube -v ~/.aws:/root/.aws  harryh00/docker-kits:alpine-k8s
```


## [Github Repository](https://github.com/harryho/docker-kits.git)

## [Docker Hub Repository](https://hub.docker.com/r/harryh00/docker-kits)
 
<file_sep>+++

title = "Unpacking"
description = "Data structure - Unpacking "
draft=true
weight=20
+++

### Data structure

#### Unpacking 

* Unpacking a Sequence into Separate Variables

```py
>>> p = (4, 5)
>>> x, y = p # x = 4, y = 5

>>> data = [ 'ACME', 50, 91.1, (2012, 12, 21) ]
>>> name, shares, price, date = data # date =  (2012, 12, 21)
>>> name, shares, price, (year, mon, day) = data # 
```

* Unpacking Elements from Iterables of Arbitrary Length

```py
def drop_first_last(grades):
    first, *middle, last = grades
    return avg(middle)
```

* Another use case. The phone_numbers variable below will always be a list, regardless of how many phone numbers are unpacked (including none).

```py
>>> record = ('Dave', '<EMAIL>', '773-555-1212', '847-555-1212')
>>> name, email, *phone_numbers = user_record
>>> phone_numbers
['773-555-1212', '847-555-1212']
>>> record = ('ACME', 50, 123.45, (12, 18, 2012))
>>> name, *_, (*_, year) = record
>>> year # 2012
```

* Combine with splitting function

```python
>>> line = 'nobody:*:-2:-2:Unprivileged User:/var/empty:/usr/bin/false'
>>> name, *fields, homedir, sh = line.split(':')
>>> sh # /usr/bin/false
```

* It can be especially useful when iterating over a sequence of tuples of varying length. And these iterables have some known component or pattern in their construction. 

```python
records = [
('foo', 1, 2),
('bar', 'hello'),
('foo', 3, 4),
]
def do_foo(x, y):
    print('foo', x, y)
def do_bar(s):
    print('bar', s)

# Only works for python 3
for tag, *args in records:
    if tag == 'foo':
        do_foo(*args)
    elif tag == 'bar':
        do_bar(*args)

```


#### Keeping the Last N Items

* keep a limited history of the last few items seen during iteration or during
some other kind of processing.

* Keeping a limited history is a perfect use for a collections.deque

```py
from collections import deque

def search(lines, pattern, history=5):
  previous_lines = deque(maxlen=history)

  for line in lines:
    if pattern in line:
      yield line, previous_lines
    previous_lines.append(line)

# Example use on a file
if __name__ == '__main__':
  with open('main.py') as f:
    for line, prevlines in search(f, 'python', 5):
      for pline in prevlines:
        print(pline, end='')
      print(line, end='')
      print('-'*20)

### Output
#     previous_lines.append(line)

# # Example use on a file
# if __name__ == '__main__':
#   with open('main.py') as f:
#     for line, prevlines in search(f, 'python', 5):
```





<file_sep>+++
date = "2017-06-07T16:56:21+11:00"
title = "Angular vs React vs Vue"
description="Angular, React, Vue as most popular JavaScript frameworks at present, we just discuss Angular 1.x, Angular 2 / 4, React 15+ (Redux), and Vue 2+ here "
+++

>Angular, React, Vue as most popular JavaScript frameworks at present, we just discuss Angular 1.x, Angular 2 / 4, React 15+ (Redux), and Vue 2+ here. There is no Angular 3, if you have not noticed before.


## Client side is a battle field

In past 6–8 years, the Restful API has been accepted as one of standard web interfaces for most web applications, solution architect can simply add REST API on the top of existing web layer or business layer to provide REST API and support multiple client devices. So the developers can continue to develop or maintain system with their favorite programming language, framework or technical stacks.

On the contrary, it is a completely different story on the client-side, there are tons of variant JavaScript frameworks emerged in the last 10 years. It is good to have more options, but it is a nightmare for web developers who are working in such battle field, because they need to try a lot of different stuff to make a decision, especially when the project schedule is tight, it makes tech lead or architect quite stressed. Something becomes much worse is when the development team try to adopt new framework for the new project, it is not easy to work out which framework we should choose.

Don’t forget there is another big risk to adopt new programming language(ES 6 or Typescript) with the new framework, as well as new development, build and test tool, if the team has no enough skills or experience. As solution architect, you need to think it through for development team, and also consider if the team can really pick it up quickly. That is why we have to compare those frameworks here before we make a decision.

## Common pitfalls  to avaoid

### Performance is not a priority

We can find lots of comparison between those frameworks, and so many of them are related to performance, programming language, design pattern, etc. Actually many web applications in the world are just small to median size web application, we don’t need to build the web application as Google, Facebook or Twitter. In my opinion, the performance of framework is the not critical benchmark, at lease it is not first priority which we need to consider if it is right for the team. Except performance, we have more concern on tech stacks, community and ecosystem involved with the framework, which have more impact on team’s productivity and system’s maintainability. 

### Cool stuff is not always the best

We have seen so many cool stuff which are finally abandoned in the past, the Silverlight is one of such examples. We shouldn’t choose new framework because it looks cool or it is the latest one. We choose the new one because it really can solve our problem, improve our productivity and quality in the long run. Don’t forget there is always some cost to adopt new things. We need to balance the cost and outcome of the technical investment, and we need to work it out if it is right time to do it.


###  Programming language is still the barrier


If we need to use new programming language, we have to evaluate with the existing development team. Even ES6 or TypeScript (TS) claims it is compatible to Vanilla JS, but when you start to look into new framework or sample project, which are coded with ES6 or TS, it still makes you so confused if you are not familiar with such syntax. It will significantly impact the efficiency of learning new framework. So there is always a learning curve, which we cannot ignore, to code something in a new programming language.


Someone complaints all those JS frameworks makes the build process much more complicated than the old web frameworks, because of the new programming language. Does it really matter? The short answer is Yes, but we are not going discuss the advantage in details here. If your team comes from .Net Web Form or Java MVC background, it would be a steep curve for the team to pick up ES6 or TypeScript and Component-based framework, not to mentioned new build and test tools.


No wonder a few .Net teams were struggling with Node stack integration on Visual Studio, especially when the team members have no Node.js experience. So we need the whole team to discuss the difficulties before we adopt new technology and framework. It is helpful to make sure the team has the same view, and it is also important to plan our training and decide how to transform development team step by step.


## The difference of those framework

> Let's look into the frameworks and list the difference of these frameworks.

* Basic tech stacks


Tech Stacks   |    Angular 1.x   | Angular 2 / 4 | React 1.5 (Redux)  | Vue 2   
----------------|-------------------| ---------------|---------------------|---------
Vanilla JS     |     Yes       |  Supported  |  Supported       | Supported
ES 6          |     1.5+          |  Supported  |  Yes            | Yes
TypeScript    |               |  Yes        |                | Supported |
MVC           |   1.2-1.4 |            | |
Component-Based |  1.5+ |  Yes   | Yes |  Yes 
Shadow DOM | | Yes | |
Virtual DOM| | | Yes | Yes
Immutable state  | | | Yes | Yes



* Yes:  Programming language which the framework uses.
* Supported: Programming language which the framework supports.


## Where to start 

For the team which comes with Web Form, with Vanilla JS background, we can start with Angular 1.x (Up to 1.4) on some small projects, or we can build something training project, because the MVC pattern is very similar to their previous coding experience.

One more thing, I have to mention is the Angular 1.x application can be built without any Node.js tools, such as Gulp, Grunt, Webpack, etc. It makes the team feel comfortable to adopt it without prior experiences. Also, it gives the team some buffer to organize the training to pick up Node.js tools for the future.

For the team which has experience of Angular 1.2 ~ 1.4, they can choose to stay on later version of Angular 1.x, e.g. Angular 1.5+, and they can start to convert coding pattern from MVC to Component-based. After that, if the team is planning to move to Angular 2 / 4, it is better to do some TypeScript training. In my view, so far the ecosystem for Angular 2 / 4 is still under development. It is a bit risky to use Angular 2 / 4 to build the real-world production. There are quite many gotchas which you have to figure out on your own.

For the team which has TypeScript or ES6 experience, they can choose what they prefer. They can spend more time on UI integration. There are a few customized UI package for bespoke framework. That is what we are going to discuss in the next.



### Responsive UI library support

> To build a real-world application, we need to integrate some popular responsive UI libraries instead of building all styles on our own. Let’s take a look the support of Bootstrap or Material-Design for different frameworks.


 UI library   |    Angular 1.x     | Angular 2 / 4 | React 1.5 (Redux)  | Vue 2   
--------------|------------------- | ---------------|------------------|---------
Bootstrap 3  | ui-bootstrap (Very Good) |            | react-bootstrap(Very Good) | VueStrap* (Very Good)
Bootstrap 4  |                          | ui-bootstrap (Alpha) | In progress | BootstrapVue (Good) |
Material Design | Materialize (Good)   | Angular Material(Basic)  |  Material-UI (Good) | Vuetify (Very Good)



* VueStrap: Please use the [Willen's fork](https://github.com/wffranco/vue-strap) for Vue 2.
* Libraries in the table above has been tested or used in some projects.


From what we can see now, the Bootstrap 4 is similar to Material-Desgin. So it is good news for developer. They just need to pick their favor, and they will always get analogical effect.


Actually there are tons of UI libraries / CSS framework available on Github, so many are platform neutral framework, i.e. It can be integreated with Angular, React or Vue. Be honest, integration is always not easy, it will take you or your team some extra effort. Keep it in mind, to integrate platform neutral framework you need to take care of dependencies and build, test tools on your own, such as webpack or yarn.

### Stable API 

Against to Angular 1.x, the Angular 2 is completely a new animal. Angular 4 comes with some breaking changes, which breaks a few Angular 2 dependencies (third parties). Since the API of Angular 4 is still under active development, we cannot use it for production. According to Angular team’s announcement, they want to fix all Angular-2’s bugs and issues in Angular 4 and keep all built-in libraries sync to Angular 4. It will take a long while to get things ready. If your project uses Angular 1.2–1.4, I’d like to suggest you to keep it, until Angular 4 is finalized.

React-Redux is much more popular than React-Flux recently, but it doesn’t means it is better than React-Flux pattern. In my opinion, React-Flux is much more straight and close to original React design. If you ready use React-Flux, you have better to stick with it.

Vue 2 comes with some breaking changes. There is migration guide for Vue 1.x to Vue 2. It doesn’t seem very different. Vue 2 is ready for production.

## How to compare  

In order to compare those frameworks properly, I use those frameworks to create a small real-world web application, which has built-in authentication support for the back-end API service, and integrated with some responsive UI framework, e.g. Bootstrap or Material-Design.

You will find there is no project built on the Angular 1.x, because my team and me have done a lot real-world application upon Angular 1.x. We know Angular 1.x, inlcuding its ecosystem is quite reliable, which you replicate any web application with this framework.

On the other hand, Angular 1.x is built with Vanilla JS, i.e. you don’t need transpiler to build Angular 1.x app, so it is a bit unfair to compare with the framework which is coded with ES 6 or TypeScript, because the build tool and setup for Angular 1.x is easier than others. I mention the Angular 1.x here to remind them, actually there is other option for the team comes from traditional MVC stacks. It would be proper way to transform the team smoothly.


### Following are the projects and related screenshots


[Angular 4 CRM](/projects/angular4-crm/)

![Screenshot-Angular4Crm](/img/ng4crm-screenshot-2.jpg)

[React CRM](/projects/react-crm/)

![Screenshot-React-Crm](/img/rrcrm-screenshot-2.jpg)

[Vue 2 CRM](/projects/vue2-crm/)

![Screenshot-Vue2crm](/img/v2crm-screenshot-2.jpg)



## Comparison of different framework

Let’s go back to projects above and take a look. Basically they are implemented almost the same features as real-world simple CRM application.


### __Features__

* Authentication & Token support for Restful API
* Customer CRUD functions
* Order CRUD functions
* Dashboard including two charts (Bar/Line/Doughnut)
* Integrate with Material Design (Angular project includes bootstrap)

### Size of source code 

|             |Angular 4 CRM   |  React Redux CRM  | Vue 2 CRM
|----------|-----------------|--------------|-------------
Dependencies |       22           |      13           |  9
Code Size    |     135KB          |     113KB         |  49KB
Working Hours       |     72 hrs         |    80 hrs         |    48hrs

* Dependencies: Any dependencies for test, distribute are excluded
* Code Size: It includes some customized CSS file, but image files are excluded
* Working Hours: The effort for learning curve has been eliminated, but R&D effort cannot be excluded. 

### Working hours  is a reflection of productivity

Firstly, I have to explain why React project took more effort than the other two projects. Comparing with React, the Angular 4 and Vue 2 are a bit new, i.e. there are more available packages or libraries for React on-line. As I mentioned before, it is not a good news. We need to try more different to figure out the pros and cons of different solutions. Unfortunately, we cannot exclude such R&D effort when we build these projects. 

According to above the dependencies, code size, we can see the project based on Vue.js is much simpler than other two projects.In my view, Vue 2 is my favor for next new project. It combines advantages of Angular and React. It also addresses some problems which we found in Angular and React.

Vue.js uses Virtual DOM, which avoids many dirty checking in Angular 1.x, and the complicated coding pattern (Observable & ReactiveJs, IMO) in Angular 2 / 4.

Vue.js makes the handling of immutable and mutable variables much easier than React. Its template is very handy and straight. It is the same as regular HTML, it is very easy to convert the mock-up HTML into Vue template, especially when you need to customize you styles. Vue’s template and directive is similar to Angular.

Vue.js is not just cool, it is elegant and simple. I am pretty sure if you have Angular or React background, you will pick it up in a couple hours or days. Once you start to use it, you won’t want to go back. Its official routing system is quite stable and easy to use. Compare with Angular-Router or React-Router, it is much more reliable.

Generally, Material-Design libraries for React is not handy as other customized version for Angular or Vue. The special coding style of JSX needs to convert all CSS and HTML into JSX format. To be honest, I am not so convinced by React’s JSX, because it is not straight as final HTML or CSS. Compare to other framework, it is a bit verbose and inconvenient. We cannot simply copy the style code from the browser’s dev tool when we debug it on the browser. i.e. You need to put more effort to make your page pretty.

Angular’s Material-Design library has very limit components. To build a real-world application you need to add another UI library to supplement the former missing components. Last but not least, the Vuetify is the best Material-Design so far we have found and tested.

Angular 4 CRM — https://github.com/harryho/ng4crm

React Redux CRM — https://github.com/harryho/react-crm

Vue 2 CRM — https://github.com/harryho/vue2crm


## Summary

Before we make any conclusion, we have to be aware the world keeps changing. Perhaps when I was writing this article, some problem of framework have been solved, or some small problem became worse and worse. We have to review the decision we made from time to time and correct them ASAP if we find the cost is overweight the outcome.

* Team with Web Form and Vanilla Js background should starts with Angular 1.4 and take some time to be familiar with Node.js tools

* Team with Vanilla Js background should start to learn ES 6 or TypeScript, since sooner or later all the browsers, including mobile devices, will support ES 6 or TypeScript.

* Teamwith ES6 /TypeScript background can choose any framework you prefer, the integration with other UI library will take you some time to make a judgement.

* Teamwith React-Flux can continue or switch to React-Redux. It maybe reduces some boiler code, but I don’t think it is a big deal.

* For the newcomer of React, I will recommand React-Redux, because it has better community support.

* In my opinion, if you continue to invest anything on Angular 2, it is a bit waste, because Angular team hopes you to move to Angular 4 as soon as possible once Angular 4 is ready for production, and they are planning to fix Angular 2’s issues in Angular 4.

* Angular 4 and its ecosystem are under active development, but please be careful if you want to use them in your production.

* Vue.js framework is a very nice one. Give a go on your next project.


## Update


Angular Material Design App — https://github.com/harryho/ng-md-app

React Redux CRM — https://github.com/harryho/react-crm

Vue 2 CRM — https://github.com/harryho/vue2crm

Angular 4 CRM — https://github.com/harryho/ng4crm (It is no longer maintained to support latest Angular)<file_sep>+++
title = "DigitialOcean: Lets Encrypt"
description="Lets Encrypt & Auto renewal"
weight=4
+++

> Here I continue to finish the web host setup. The last step of web host setup is to add SSL certificate for each site

### Lets Encrypt

To enable HTTPS on your website, you need to get a certificate (a type of file) from a Certificate Authority (CA). [Let’s Encrypt](https://letsencrypt.org/) is a CA. In order to get a certificate for your website’s domain from Let’s Encrypt, you have to demonstrate control over the domain. With Let’s Encrypt, you do this using software that uses the ACME protocol, which typically runs on your web host.


#### Installing Certbot

* The first step to using Let’s Encrypt to obtain an SSL certificate is to install the Certbot software on your server.

* Certbot is in very active development, so the Certbot packages provided by Ubuntu tend to be outdated. However, the Certbot developers maintain a Ubuntu software repository with up-to-date versions, so we’ll use that repository instead.


```bash
#  First, add the repository
sudo add-apt-repository ppa:certbot/certbot

# Install Certbot’s Nginx package with apt
sudo apt install python-certbot-nginx
```

* In order to configure SSL for Nginx, we need to verify some of Nginx’s configuration.

### Confirming Nginx’s Configuration

* In the previous setup, the site for __domain-one.com__ has been up and running in the droplet. 
* Open the server block file of __domain-one.com__  via any text editor:

        sudo vi /etc/nginx/sites-available/example.com

* Find the existing __server_name__ line in the file

        server_name domain-one.com www.domain-one.com;

* If you change the server name, you need to re-run the test and reload nginx

        sudo nginx -t
        sudo systemctl reload nginx

Certbot can now find the correct server block and update it.

Next, let’s update the firewall to allow HTTPS traffic.

### Allowing HTTPS Through the Firewall

* UFW configuration has been done in the prevous setup. It is good to confirm the UFW configuration. 

* You can see the current setting by typing:

        sudo ufw status
        Output
        Status: active

        To                         Action      From
        --                         ------      ----
        OpenSSH                    ALLOW       Anywhere
        Nginx Full                 ALLOW       Anywhere
        OpenSSH (v6)               ALLOW       Anywhere (v6)
        Nginx Full (v6)            ALLOW       Anywhere (v6)

* If the 'Nginx Full' is not allowed in your UFW, please run commands below to enable it.

        sudo ufw allow 'Nginx Full'
        sudo ufw delete allow 'Nginx HTTP'

* Next, let’s run Certbot and fetch our certificates.


### Obtaining an SSL Certificate

Certbot provides a variety of ways to obtain SSL certificates through plugins. The Nginx plugin will take care of reconfiguring Nginx and reloading the config whenever necessary. To use this plugin, type the following:

    sudo certbot --nginx -d example.com -d www.example.com

This runs certbot with the --nginx plugin, using -d to specify the names we’d like the certificate to be valid for.

If this is your first time running certbot, you will be prompted to enter an email address and agree to the terms of service. After doing so, certbot will communicate with the Let’s Encrypt server, then run a challenge to verify that you control the domain you’re requesting a certificate for.

If that’s successful, certbot will ask how you’d like to configure your HTTPS settings.
    
    # Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
    # -------------------------------------------------------------------------------
    # 1: No redirect - Make no further changes to the webserver configuration.
    # 2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
    # new sites, or if you're confident your site works on HTTPS. You can undo this
    # change by editing your web server's configuration.
    # -------------------------------------------------------------------------------
    # Select the appropriate number [1-2] then [enter] (press 'c' to cancel):
    # Select your choice then hit ENTER. The configuration will be updated, and Nginx will 
    # reload to pick up the new settings. certbot will wrap up with a message telling 
    # you the process was successful and where your certificates are stored:
    # ...
    # ...
    

### Renew certificate

Let’s Encrypt’s certificates are only valid for ninety days. This is to encourage users to automate their certificate renewal process.

#### Verifying Certbot Auto-Renewal

The certbot package we installed takes care of this for us by adding a renew script to `/etc/cron.d`. This script runs twice a day and will automatically renew any certificate that’s within thirty days of expiration.

To test the renewal process, you can do a dry run with certbot:

    sudo certbot renew --dry-run


#### Use Cron to renew certificate

Here I just add the cronjob to system crontab `/etc/crontab`

    # Lets encrypt cronjob - start from 1st April 2020
    0  0    1 */3 * root  /usr/bin/certbot renew >> /var/log/letsencrypt/renew.log
<file_sep>+++
date = "2014-01-10T14:59:31+11:00"
title = "CentOS 6/7 Multi-Boot Setup"
description = "CentOS 6/7 Multi-Boot note"
draft = false
+++


## Bootable usb preparation

> Download dvd iso from url or torrent
> Use Win32 Image Writer to create usb. ( Bootice is useful tool to reformat the USB as origin )

## Install CentOS on virtual machine

* Before you install, you are better to backup anything on the device which you are going to install, and chcek your internet is working properly. 

* CentOS provide a friendly and nice installation process. If you choose VMWare or VirtulBox as machine, you can open [installation steps](http://www.tecmint.com/centos-7-installation) on your browser or use ipad /tablet to access this page. You just need to follow the instruction step by step, it will be done within an hour or more(it varies in computing power of PC).


## Prepare VM for CentOS

* Create a new virtual machine. 
    * Type in the name of vm. e.g. CentOS
    * Select the type of Linux
    * You can choose RedHat(32/64 bit ) or something else. It doesn't matter. We don't use any built-in xxx.iso files from VirtualBox. Then click `Next`
    * Select the memory size for the CentOS. It is up  to you. I prefer 4GB, but 2GB is necessary. And then click `Next`.
    * Select "Create a virtual hard drive now", and then click `Next`.
    * Select defaut VDI, then click `Next`.
    * Select "Dynamically allocated", then click `Next`.
      * Choose the location of CentOS. e.g. c:\vbox\centos\centos.vdi. Select the size of VDI file. At least 10G. I'd like to select 20 or more. Then a virtual machine of CentOS is created. 
      * Choose the location of CentOS. e.g. c:\vm\centos\centos.vmdk. Select the size of VMDK file. At least 10G. I'd like to select 20 or more. Then a virtual machine of CentOS is created. 
 
 * Config the CentOS hardware setting on VirtulBox or VMWare. 

    **VirtualBox**

    * On VirtualBox toolbar, there is a `Start` button. Click `Start`, then go the Storage item. 
    * Under the Storage Tree section, there is Empty CD icon. Click the Empty icon.     
    * Under the Attributes section, click the CD icon at the end of the dropdown list of CD/DVD Drive. Choose the Unbuntu iso file which you download from Unbuntu.org. Click `OK`.
    * Leave all the other setting as default. Click the `Start`button on the toolbar. 

    **VMWare**

    * On VMware, you can find CD/DVD button on the tab page of new virtual machine. 
    * Click the CD button at the end of the dropdown list of CD/DVD Drive. Choose the Unbuntu iso file which you download from Unbuntu.org. Click `OK`.
    * Leave all the other setting as default. Click the `Power on this virtual machine` option on the tab page. 
 
## Config CentOS default setting
 
 After the CentOS starts, we can config the default setting of CentOS. Don't be panic, the configuration envrionment is very nice. You don't need to type any command line so far.
 
* Prepare the installation.
    * Choose the language of CentOS. Then click Continue.
    * On the Installation Summary screen, there is warning icon attached to the hard drive icon, which is under SYSTEM section with a lable "INSTALLATION DESTINATION". Click that label.  
    * You will see the partition is already done automatically. I'd like you to leave it as until you are familar with the CentOS. Then click Done button on the header.
    * Leave the SOFTWARE SELECTION as Minimal Install; NETWORK & HOSTNAME as Not connected. Then click Begin Installation button.

* Setup account
    * Setup password of root account. Please remember the password. If you forget it you need to reset the root's password. To do that you need to do a few things which depends on the CentOS version. As a beginner, please don't make it too complicated. 
    * Create your account. e.g. harryporter Mark your account as administrator to save some effort. Please remember your password and don't make it too complicated.'
    * The progress of installation is complete. Click Finish configuration button.
     w$w4 d. After a few seconds, you will the Reboot button. Then click it. 

## Manage CentOS packages and software

* there are two management tools rpm and yum. To make it easy, we just talk about yum. It is a very handy tool. Comparing with the windows built-in program management tool, it is much powerful than that. It provides necessary functions for admin. If you need to maintain the Linux server, then you will use it in your daily task. 
* Use `man yum` to take a look the description of yum. You do not need to understand all usage of yum. Just have an overview is enough. 
* There are a few useful and common yum commands.  

```bash
yum list installed | less
yum search <pacakge_name>
yum grouplist
```


### Install CentOS on PC or laptop

## Setup network ( via cable )

If you install a minimal version without network configuration, you will find you can not ping public domain. Here I am going to show you how to setup the connection. 
 
* You can use `ip a` command to check the status of all network interface. 
* You will see all the state of interfaces would be DOWN OR UNKNOWN.
 
```bash

## lo ****  qdisc pfifo_fast noqueue UNKNOWN
## eth0 **** qdisc pfifo_fast state DOWN

```
 
* Use `## ifup eth0` to start the ech0 then you can access internet. It is a bit tedious to start the network service every time we reboot the system. Next step we will setup network service to start automatically after the system is up. 
* There is a configuration file which can help you setup internet connection after the startup. Ususally the configuration is under this path `/etc/sysconfig/network-scripts`, and file name would `ifcfg-eth<*>`. So we check the real file name at first. 
 
```bash
ls /etc/sysconfig/network-scripts/ifcfg-*

## Following are sample of files which will sit in your system. 
/etc/sysconfig/network-scripts/ifcfg-eth0
/etc/sysconfig/network-scripts/ifcfg-lo
```

Now we need to update this file via nano or vi. If you have no any experience of vi, I recommended an interactive online tutorial for you. Just 20 mins, you can master basic vi usage. 
http://www.openvim.com/tutorial.html

We use vi to open the config file.
```bash
## vi /etc/sysconfig/network-scripts/ifcfg-eth0
```

You will a setting below.
ONBOOT=no

You just need to update it to ONBOOT=yes, then save it and reboot CentOS to test the result.
```bash
## reboot
```

After you reboot and login CentOS, you can use ping to test if your system can access internet, then configuration is updated successful.


### Setup Wifi 

* Setup Wifi during the installation. After the installation, you will find the Wifi is not available on Cent OS
* Mount the DVD or iso file
* use yum to install the NetworkManager-wifi package 

```bash
yum --disablerepo=\* install /path/to/dvd/Packages/NetworkManager-wifi* 
```

### Multiple boot system ( Fedora, CentOS, Redhat )

## Partition setup for multiple OS installation 

> Fedora, CentOS and Redhat share the almost the same installation process. 
> Click on the `Installation Destination` icon to change this to custom partitioning
> Under the Other Storage Options, choose I will configure partitioning then click Done
> Following is sample of partition setup of multiple boot system.


```ini
sda
  +----sda1        nfts               500M         Windows recovery
  +----sda2        efi       /boot    100M         grub2 , Windows boot manager
  +----sda3                  /        10M            
  +----sda4        ntfs      /      40000M         Window 7/8/10
  +----sda5                  swap   <Double size of your RAM size>   
  +----sda6        ext4      /      20000M         Ubuntu 14 desktop
  +----sda7        ext4      /      20000M         Fedora 20 desktop           
  +----sda8        ext4      /      20000M         CentOS 6 desktop
  +----sda9        ext4      /      20000M         OpenSuse desktop
  + ...

```

## Troubleshooting

### ifconfig not found in CentOS minimal server

Use command `ip`

```
ip addr
ip -s link
```

### Enable Network (Non-wifi) onboot after minimal installation

If you cannot ping any domain, use `dhclient -v` to check if the internet is available. 

Setup the network enabled onboot

```
## cd /etc/sysconfig/network-scripts/ 
## sed -i -e 's@^ONBOOT="no@ONBOOT="yes@' ifcfg-e.xx.xxx
``` 

### Boot CentOS in terminal

```
## cat /etc/inittab
## systemctl get-default 
graphic.target
## systemctl set-default multi-user.target
```

### Fedora boot error

* Please check the grub.cfg if you get booting error
* You can try following command to boot Fedora from Grub menu

```bash
linux /boot/vmluz-x.x.x-x.x.x
initrd /boot/intrd-plymouth.img
```

<file_sep>IP_LOG=ip.log
LOG=aws_sg.log

echo $SG_LIST

show_sg_ids() {
    aws ec2 --profile $PROFILE describe-security-groups \
        --output json \
        --filters "Name=group-name,Values=*Bastion*" \
        --query 'SecurityGroups[*].{Name:GroupName,ID:GroupId,permissions:IpPermissions[*]}' | jq
}

get_perm() {
    PROFILE=$1
    # echo $PROFILE | tee -a $LOG
    if [[ $PROFILE == "uss" ]]; then
        PERM='[{"IpProtocol":"tcp","FromPort":22,"ToPort":22,"IpRanges":[{"CidrIp":"IP_ADDRESS/32"}]}]'
    elif [[ $PROFILE == "ad1" ]]; then
        PERM='[{"IpProtocol":"tcp","FromPort":3389,"ToPort":3389,"IpRanges":[{"CidrIp":"IP_ADDRESS/32"}]}]'
    fi

    echo $PERM
}

get_desc() {
    PROFILE=$1

    if [[ $PROFILE == "uss" ]]; then
        DESC='[{"IpProtocol":"tcp","FromPort":22,"ToPort":22,"IpRanges":[{"CidrIp":"IP_ADDRESS/32","Description":"Harry"}]}]'
    elif [[ $PROFILE == "ad1" ]]; then
        DESC='[{"IpProtocol":"tcp","FromPort":3389,"ToPort":3389,"IpRanges":[{"CidrIp":"IP_ADDRESS/32","Description":"Harry"}]}]'
    fi
    # echo $PROFILE | tee -a $LOG
    echo $DESC
}

update_sg() {
    PROFILE=$1
    SGID=$2
    OIP=$3
    NIP=$4

    echo $PROFILE $SG $OIP $NIP | tee -a $LOG

    PERM=$(get_perm $PROFILE)
    DESC=$(get_desc $PROFILE)

    OLD_PERM=${PERM/"IP_ADDRESS"/$OIP}
    NEW_PERM=${PERM/"IP_ADDRESS"/$NIP}
    NEW_DESC=${DESC/"IP_ADDRESS"/$NIP}

    echo $OLD_PERM | tee -a $LOG

    echo $NEW_PERM | tee -a $LOG
    echo $NEW_DESC | tee -a $LOG

    aws ec2 --profile $PROFILE \
        revoke-security-group-ingress \
        --group-id $SGID --ip-permissions $OLD_PERM

    aws ec2 --profile $PROFILE \
        authorize-security-group-ingress \
        --group-id $SGID \
        --ip-permissions $NEW_PERM

    aws ec2 --profile $PROFILE \
        update-security-group-rule-descriptions-ingress \
        --group-id $SGID --ip-permissions $NEW_DESC

    aws ec2 --profile $PROFILE \
        describe-security-groups \
        --output json \
        --group-ids $SGID | jq

}

update_ad1_sg() {
    OIP=$1
    NIP=$2

    AD1_SG_LIST=(
        sg-0ad548ef15af34a70 # AD1_Dev_Web_AZ2_SG
        sg-0a6169737f7d9f004
        sg-016fe912b6508922f
        sg-357d1c4c
        sg-afb1aac8
        sg-dd1470a4
        sg-e9abf48e
        sg-2ea3fc49          # NSW PROXY
        sg-015675cf1a33c7ac8 # NSW UAT
        sg-0a16987f96e469902 # NSW1-PROD-BASTION-SG
    )
    echo " :::::::::::: PROFILE - ad1 :::::::::::: " | tee -a $LOG
    for SG in "${AD1_SG_LIST[@]}"; do
        update_sg ad1 $SG $OIP $NIP
    done
    show_ips ad1
}

update_uss_sg() {
    OIP=$1
    NIP=$2

    USS_SG_LIST=(
        sg-01964d6e858a0ca06
        sg-021143c587a1cc824
        sg-024d069fa41797225
        sg-04e182d3cd00aab71
        sg-30865449
        sg-3736e74e
        sg-878351fe
        sg-028138035f54e1019
        sg-04e182d3cd00aab71
        sg-0fc6ba78da9c3db99
        sg-d771cbae
        sg-5805bf3f # FTP

    )
    echo " :::::::::::: PROFILE - uss :::::::::::: " | tee -a $LOG
    for SG in "${USS_SG_LIST[@]}"; do
        update_sg uss $SG $OIP $NIP
    done
    show_ips uss
}

show_ips() {
    PROFILE=$1
    aws ec2 --profile $PROFILE \
        describe-security-groups \
        --output json \
        --query 'SecurityGroups[*].{Name:GroupName,ID:GroupId,permissions:IpPermissions[*]}' | grep -i "Harry" -C 2
}

OIPS=(
  192.168.127.12
)

main() {
    echo 'Start...' $(date) | tee -a $LOG

    # echo $PROFILE  | tee -a $LOG

    for OLD_IP in ${OIPS[@]}; do
        NEW_IP=$(curl ifconfig.me)

        echo Old IP $OLD_IP | tee -a $LOG
        echo New IP $NEW_IP | tee -a $LOG

        # AD1 session
        update_ad1_sg $OLD_IP $NEW_IP

        ## USS session
        update_uss_sg $OLD_IP $NEW_IP
    done
    echo "DONE $(date) !!!!!!!!!! " | tee -a $LOG

}

main
    
# 172.16.31.10
# 172.16.58.3
# 172.16.58.3
# 172.16.58.3
# 172.16.58.3
# 172.16.31.10
# 14.203.89.47<file_sep>+++

title = "Inheritance & Polymorphism"
description = "Inheritance & Polymorphism"
weight=5
+++

## Inheritance & Polymorphism


* Specify single inheritance by putting a base class in parentheses after defining a class's name
* Subclasses have all of the methods of their base class
* It's often best to explicitly call a base class initializer from a subclass's initializer
* If a class with a single base class doesn't define an initializer, the base class's initializer will be called automatically on construction
* Python treats `__init__()` like any other method
* Base class `__init__()` is not called if overridden
* Use `super()` to call base class `__init__()`
* `isinstance()` takes an object as its first argument and a type as its second
* `isinstance()` determines if its first argument is an instance of the second argument, or any subclass of the second argument
* `isinstance()` can accept a tuple of types as its second argument, in which it returns True if the first argument is of any of those types
* Checking for specific types is rare in Python and is sometimes regarded as bad design
* `isinstance()` determines if its first argument is a direct or indirect subclass of, or the same type as, the second argument
* Multiple inheritance means having more than one direct base class
* You declare multiple base classes with a comma-separated list of class names in parentheses after a class's name in a class definition
* A class can have as many base classes as you want
* Python uses a well-defined "method resolution order" to resolve methods at runtime
* If a multiply-inheriting class defines no initializer, Python will automatically call the initializer of its first base class on construction
* `__bases__` is a tuple of types on a class object which defines the base classes for the class
* `__bases__` is in the same order as in the class definition
* `__bases__` is populated for both single and multiple inheritance
* Method resolution order defines the order in which Python will search an inheritance graph for methods
* MRO is short for Method Resolution Order
* MRO is stored as a tuple of types in the `__mro__` attribute of a class
* The `mro()` method on type objects returns the contents of `__mro__` as a list
* To resolve a method, Python uses the first entry in a class's MRO which has the requested method
* MRO is dependent on base class declaration order
* MRO is calculated by Python using the C3 algorithm
* MRO honors base-class ordering from class definitions
* MRO puts subclasses before base classes
* The relative order of classes in an MRO is consistent across all classes
* It is possible to specify an inconsistent base class ordering, in which case Python will raise a TypeError when the class definition is reached
* `super()` operates by using the elements in an MRO that come after some specified type
* `super()` returns a proxy object which forwards calls to the correct objects
* There are two distinct types of `super()` proxies, bound and unbound
* Unbound `super()` proxies are primarily used for implementing other Python features
* Bound proxies can be bound to either class objects or instances
* Calling `super()` with a base-class and derived-class argument returns a proxy bound to a class
* Calling `super()` with a class and an instance of that class returns a proxy bound to an instance
* A `super()` proxy takes the MRO of its second argument (or the type of its second argument), finds the first argument in that MRO, and uses everything after it in the MRO for method resolution
* Since class-bound proxies aren't bound to an instance, you can’t directly call instance methods that they resolve for you
* However, classmethods resolved by class-bound proxies can be called directly
* Python will raise a TypeError if the second argument is not a subclass or instance of the first argument
* Inappropriate use of `super()` can violate some design constraints * Calling `super()` with no arguments inside an instance method produces an instance-bound proxy
* Calling `super()` with no arguments inside a classmethod produces a class-bound proxy
* In both cases, the no-argument form of `super()` is the same as calling `super()` with the method's class as the first argument and the method's first argument as the second
* Since `super()` works on MROs and not just a class's base classes, class can be designed to cooperate without prior knowledge of one another
* The class object is at the core of Python's object model
* object is the ultimate base class for all other classes in Python
* If you don't specify a base class for a class, Python automatically uses object as the base
* Because object is in every class's inheritance graph, it shows up in every MRO.
* object provides hooks for Python's comparison operators
* object provides default `__repr__()` and `__str__()` implementations
* object implements the core attribute lookup and management functionality in Python
* Inheritance in Python is best used as a way to share implementation


### Explanation with example


#### Example code  

* The code below demonstrates the weird `super()` in Python

    ```python
    from pprint import pprint as pp

    class Parent(object):
        name = 'Parent'

        @classmethod
        def do_otherthing(self):
            print('This is from Parent.  {} do_otherthing'.format(self.name))
            ## print('This is from Parent. do_otherthing {}'.format(self.name))

        def do_something(self):
            print('This is from Parent. The name is {}'.format(self.name))
    
    class Child(Parent):
        name = 'Child'

        @classmethod
        def do_otherthing(self):
            print('This is from Child. {} do_otherthing'.format(self.name))
            ## print('This is from Child. do_otherthing {}'.format(self.name))

        def do_something(self):
            print("This is from Child. The name is {} ".format(self.name))
    
    class OtherChild(Parent):
        name = 'OtherChild'
    
        def do_something(self):
            print("This is from OtherChild. The name is {} ".format(self.name)) 
    
    class OtherOtherChild(Parent):
        name = 'OtherOtherChild'
    
        def do_something(self):
            print("This is from OtherOtherChild. The name is {} ".format(self.name))
    
    class GrandChild(Child, OtherChild, OtherOtherChild):
        name = 'GrandChild'

        @classmethod
        def do_otherthing(self):
            print('This is from GrandChild. {} do_otherthing'.format(self.name))

        def do_something(self):
            print("This is from GrandChild. The name is {}".format(self.name))


    if __name__ == "__main__":
        pp(Child.__mro__)
        pp(GrandChild.__mro__)

        c = Child()
        c.do_something()
        gc = GrandChild()
        gc.do_something()

        print('Class bound super()')
        super(Child, Child).do_otherthing()
        super(Child, GrandChild).do_otherthing()
        super(GrandChild, GrandChild).do_otherthing()

        print('Instance bound super()')
        super(Child, gc).do_something()
        super(OtherChild, gc).do_something()
        super(GrandChild , gc).do_something()

    ### test result 

    #(<class '__main__.Child'>, 
    #<class '__main__.Parent'>, 
    #<class 'object'>)
    #(<class '__main__.GrandChild'>,
    ## <class '__main__.Child'>,
    ## <class '__main__.OtherChild'>,
    ## <class '__main__.OtherOtherChild'>,
    ## <class '__main__.Parent'>,
    ## <class 'object'>)
    ## This is from Child. The name is Child 
    ## This is from GrandChild. The name is GrandChild
    ## Class bound super()
    ## This is from Parent.  Child do_otherthing
    ## This is from Parent.  GrandChild do_otherthing
    ## This is from Child. GrandChild do_otherthing
    ## Instance bound super()
    ## This is from OtherChild. The name is GrandChild 
    ## This is from OtherOtherChild. The name is GrandChild 
    ## This is from Child. The name is GrandChild 


    ## Multi-inheritance in Python is very different from 
    ## other OO language like C++, C#, Java
    ## Above result make me surprised a the first time, 
    ## but after I check the MRO then I can understand 
    ## why many veterans suggest to avoid multi-inheritance. 
    ## Its `super` "Magic" really confuses many people.

    ```

#### Explanation with break down

* Let's break down the how the super works with sample code

##### Case 1: Parent and child

* Case 1:    
    * code : `super(Child, Child).do_otherthing()`
    * `super` takes the MRO of its second argument `Child`

        ```
        <class '__main__.Child'>, 
        <class '__main__.Parent'>, 
        <class 'object'>
        ```
    
    * `super` finds the first argument `Child` in that MRO, and uses everything after it in the MRO for method resolution
    * `super` uses the method from `Parent` which is after `Child`, and bind the `Child` class object
    
##### Case 2: Grandparent and grandchild

* Case 2: 

    * code : `super(GrandChild, GrandChild).do_otherthing()`
    * `super` takes the MRO of its second argument `GrandChild`

        ```
        <class '__main__.GrandChild'>,
        <class '__main__.Child'>,
        <class '__main__.OtherChild'>,
        <class '__main__.OtherOtherChild'>,
        <class '__main__.Parent'>,
        <class 'object'>
        ```

    * `super` finds the first argument `GrandChild` in that MRO, and uses everything after it in the MRO for method resolution
    * `super` uses the method from `Child` which is after `GrandChild`, and bind the `GrandChild` class object


##### Case 3: Grandparent, children & grandchild

* Case 3:   

    * code : `super(OtherChild, gc).do_something()`
    * `super` takes the MRO of the type `GrandChild` of its second argument `gc` 

        ```
        <class '__main__.GrandChild'>,
        <class '__main__.Child'>,
        <class '__main__.OtherChild'>,
        <class '__main__.OtherOtherChild'>,
        <class '__main__.Parent'>,
        <class 'object'>
        ```

    * `super` finds the first argument `OtherChild` in that MRO, and uses everything after it in the MRO for method resolution
    * `super` uses the method from `OtherOtherChild` which is after `OtherChild`, and bind the `gc` class object

<file_sep>+++
date = "2017-10-11T11:59:31+11:00"
title = "Try Minikube"
description="Test Minikube on virtual machine"

+++


## Prerequisites


* Install KVM or VirtualBox
* Install Ubuntu / Debian on KVM or VirtualBox
* Install Minikube 



## Check K8s version and config 

    kubectl version
    kubectl config
    kubectl cluster-info


## Get / Describe command

    kubectl get ndoes
    kubectl get pods
    kubectl get deployments
    kubectl get services
    
    kubectl describe pods


## Deploy hello-world node demo app    

* Deploy a demo app

    ```bash
    kubectl run kubernetes-bootcamp --image=gcr.io/google-samples/kubernetes-bootcamp:v1 --port=8080

    # View deployments and pods
    kubectl get deployments
    kubectl get pods
    ```

* Pods that are running inside Kubernetes are running on a private, isolated network. By default they are visible from other pods and services within the same kubernetes cluster, but not outside that network. 
* The kubectl command can create a proxy that will forward communications into the cluster-wide, private network. 


    ```bash
    # Create a proxy from another terminal
    kubectl proxy

    # Test it from original termianl
    curl http://localhost:8001/version
    ```

* Get pod name

    ```bash
    export POD_NAME=$(kubectl get pods -o go-template --template \
     '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')
    echo Name of the Pod: $POD_NAME
    kubectl logs $POD_NAME
    ```

* Execute command on containter

    ```bash
    kubectl exec $POD_NAME env
    kubectl exec $POD_NAME bash
    kubectl exec $POD_NAME curl localhost:8080
    ```

* Scale demo app with replica

    ```bash
    # kubernetes-bootcamp   deployment is the same as above
    kubectl get pods -o wide
    kubectl describe deployments/kubernetes-bootcamp    
    kubectl scale deployments/kubernetes-bootcamp --replicas=2

    # Get NodePort
    export NODE_PORT=$(kubectl get services/kubernetes-bootcamp \
        -o go-template='{{(index .spec.ports 0).nodePort}}')
    echo NODE_PORT=$NODE_PORT

    # Test load balance
    curl $(minikube ip):$NODE_PORT

    ```




<file_sep>+++
date = "2016-08-11T11:59:31+11:00"
title = "Git Practices"
description="Useful Git commands  & practices for repository management"
+++

## Push existing repository to remote

* Update the remote git url

        $ git remote set-url origin [your_new_repo_url] 

* Push it remote branch master/main

        $ git push -uf origin [master/main]
## Create a new branch with git 


### Create the branch locally

*  Create the branch on your local machine and switch in this branch
        
        $ git checkout -b [name_of_your_new_branch]
        
### Push the branch 

* Push the branch on git-repository (Github, Bitbucket)

        $ git push origin [name_of_your_new_branch]

When you want to commit something in your branch, be sure to be in your branch.



## Manage branches (Push, Fetch & Merge)

List all branches

    $ git branch

Add a new remote for your branch :

    $ git remote add [name_of_your_remote]

Push changes from your commit into your branch :

    $ git push [name_of_your_new_remote] [name_of_your_branch]

Update your branch when the original branch from official repository has been updated :

    $ git fetch [name_of_your_remote]

Then you need to apply to merge changes, if your branch is derivated from develop you need to do :

    $ git merge [name_of_your_remote]/develop

## Delete branch    

Delete a branch on your local filesystem :

    $ git branch -d [name_of_your_new_branch]

To force the deletion of local branch on your filesystem :

    $ git branch -D [name_of_your_new_branch]

Delete the branch on github :

    $ git push origin :[name_of_your_new_branch]

Compare two branch:

    $ git diff [name_of_branch1]..[name_of_branch2]

---

## Branch merge


### Fast-Forward Merge

Our first example demonstrates a fast-forward merge. The code below creates a new branch, adds two commits to it, then integrates it into the main line with a fast-forward merge.

```bash
## Start a new feature
$ git checkout -b new-feature master

## Edit some files    
$ git add <file>
$ git commit -m "Start a feature"

## Edit some files    
$ git add <file>
$ git commit -m "Finish a feature"

## Merge in the new-feature branch    
$ git checkout master
$ git merge new-feature
$ git branch -d new-feature
```

This is a common workflow for short-lived topic branches that are used more as an isolated development than an organizational tool for longer-running features.

Also note that Git should not complain about the git branch -d, since new-feature is now accessible from the master branch.

### 3-Way Merge

The next example is very similar, but requires a 3-way merge because master progresses while the feature is in-progress. This is a common scenario for large features or when several developers are working on a project simultaneously.

```bash
## Start a new feature
$ git checkout -b new-feature master

## Edit some files
$ git add <file>
$ git commit -m "Start a feature"

## Edit some files
$ git add <file>
$ git commit -m "Finish a feature"

## Develop the master branch
$ git checkout master

## Edit some files
$ git add <file>
$ git commit -m "Make some super-stable changes to master"

## Merge in the new-feature branch
$ git merge new-feature
$ git branch -d new-feature
```

### Rebase for merge

Problem: You are working with a few experienced devs constantly improving an online shooping site. After you complete the first assignment and ready to commit to master you find someone merged a change that affects or overlaps with the ones you made, and it could lead to bugs in the online-shoppping website. 

Solution: Situations like these are a big example of when you'd want to rebase. Let's say when you created your branch off of the master branch, the master branch was on commit No. 1. Every commit in your branch was put on top of commit #1. When you're ready to merge your branch to master, you find  other developers have some changes and the most recent commit is commit No. 4. **Rebasing is taking all your branch's commits and adding them on top of commit No. 4 instead of commit No. 1.** If you consider commit No. 1 as the "base" of your branch, you're changing that base to the most recent one, commit No. 4. Hence why it's called rebasing!

#### Rebase with conflict 

```bash
git rebase master

# When there is conflict, the rebase will pause. 
# You have to manually solve the conflict
# Add the resolved files to stage and commit it
git add <Resolved-File>
git commit 

# Conttinue the rebase process
git rebase --continue
```

#### Rebase interactively

Rebase to master branch 

    $ git rebase -i master 

Rebase with fixup and squash

* squash (s for short), which melds the commit into the previous one (the one in the line before)
* fixup (f for short), which acts like “squash”, but discards this commit’s message

    ```bash
    # Commit your changes
    $ git add. 
    $ git commit 


    $ git rebase  -i master
    # Update the prop up  editor
    # fixup <COMMIT-ID>
    # squahs   <COMMIT-ID> <Message>
    ```

Rebase with autosquash

    $ git rebase -i --autosquash master
    

## Git reword 

Reword the last commit message. The command below will open an editor to let you change previous commit message

    $ git commit --amend   

Reword the last commit message and author. 

    $ git commit --amend   --author="Other author <<EMAIL>>"


## Git Submodule

Add other repository into your existing project as submodule

    $ git submodule add <Git-Repository-URL>  <your-repo-folder>

Add other repository into your existing project as submodule under specific location 

    $ git submodule add <Git-Repository-URL>  <your-specific-location>/<your-repo-folder>

Add other repository into your existing project for specific branch

    $ git submodule add <Git-Repository-URL>  -b <branch-name> <your-repo-folder>    

Initialize and update submoule 

    $ git submodule init
    $ git submodule update 


## Git tag

Listing the existing tags

    $ git tag -l

Creating Tags

Git supports two types of tags: lightweight and annotated. IMO, I always prefer the annotated tag, because the tag is supposed to mark milestone in the repository's history.

Annotated Tags

    $ git tag -a v1.1 -m "release version 1.1"

Lightweight tags

    $ git tag -a v1.1-lw

Push tags

    $ git push origin v1.1

Delete a tag (local)

    $ git tag -D v1.1

Delete a tag (remote)

    $ git push --delete origin v1.1

<file_sep>+++


date = "2016-03-04T14:59:31+11:00"
title = "KVM Notes"
draft = true
+++




Installl Virtual Machine 


```
sudo virt-install \
--virt-type=kvm \
--name centos7 \
--ram 2048 \
--vcpus=2 \
--os-variant=centos7.0 \
--virt-type=kvm \
--hvm \
--cdrom=/var/lib/libvirt/boot/CentOS-7-x86_64-Minimal-1804.iso \
--network=bridge=virbr0,model=virtio \
--network=bridge=br0,model=virtio \
--graphics vnc \
--disk path=/var/lib/libvirt/images/centos7.qcow2,size=40,bus=virtio,format=qcow2
```

/var/lib/libvirt/boot/CentOS-7-x86_64-Minimal-1804.iso<file_sep>+++
title = "Azure: RBAC - 2"
weight = 1
description="Difference - Classic subscription, Azure Roles & Azure AD Roles"

+++

## Roles

- Classic subscription administrator roles
- Azure roles
- Azure Active Directory (Azure AD) roles

### History

> When Azure was initially released, access to resources was managed with just three administrator roles: Account Administrator, Service Administrator, and Co-Administrator. Later, Azure role-based access control (Azure RBAC) was added. Azure RBAC is a newer authorization system that provides fine-grained access management to Azure resources. Azure RBAC includes many built-in roles, can be assigned at different scopes, and allows you to create your own custom roles. To manage resources in Azure AD, such as users, groups, and domains, there are several Azure AD roles. To manage resources in Azure AD, such as users, groups, and domains, there are several Azure AD roles.

### High-level view 

{{<mermaid>}}
graph TB
    Root(Global Admin,User Access Admin-elevated access)
    AADT(Azure Active Directtory Tenent)
    RMG(Root Management Group)
    MG(Management Group)
    Sub(Subscriptions, Azure Account, Account Admin)
    RG(Resource Group)
    R(Resource)
    AADT --> Root
    Root --> RMG
    GAAA([Global_Admin, Application_Admin])
    OCRU([Owner,Contributor, Reader,User_Acess_Admin])
    subgraph Azure_AD_Roles
        GAAA
        AADT
    end
    Root;
    subgraph Azure_Roles
        OCRU
        RMG --> MG
        MG --> Sub
        subgraph Classic_Subscription_Admin_Roles
            Sub --> RG
            RG --> R
        end
    end
{{</mermaid >}}


### Classic subscription administrator roles

Account Administrator, Service Administrator, and Co-Administrator are the three classic subscription administrator roles in Azure. Classic subscription administrators have full access to the Azure subscription. 

Classic subscription administrators have full access to the Azure subscription. 

#### Account Administrator
* per Azure account	
* Manage billing in the Azure portal 
* Manage all subscriptions in an account
* Create new subscriptions
* Cancel subscriptions
* Change the billing for a subscription
* Change the Service Administrator
* Conceptually, the billing owner of the subscription.
* The Account Administrator has no access to the Azure portal.

#### Service Administrator	
* 1 per Azure subscription	
* Manage services in the Azure portal
* Cancel the subscription
* Assign users to the Co-Administrator role
* By default, for a new subscription, the Account Administrator is also the Service Administrator.
* The Service Administrator has the equivalent access of a user who is assigned the Owner role at the subscription scope.
* The Service Administrator has full access to the Azure portal.

#### Co-Administrator	
* 200 per subscription	
* Same access privileges as the Service Administrator, but can’t change the association of subscriptions to Azure directories
* Assign users to the Co-Administrator role, but cannot change the Service Administrator
* The Co-Administrator has the equivalent access of a user who is assigned the Owner role at the subscription scope.

### Azure account and Azure subscriptions

An Azure account represents a billing relationship. An Azure account is a user identity, one or more Azure subscriptions, and an associated set of Azure resources. The person who creates the account is the Account Administrator for all subscriptions created in that account. That person is also the default Service Administrator for the subscription.

Azure subscriptions help you organize access to Azure resources. They also help you control how resource usage is reported, billed, and paid for.

Each subscription can have a different billing and payment setup, so you can have different subscriptions and different plans by office, department, project, and so on. 

### Azure roles

Azure RBAC is an authorization system built on Azure Resource Manager that provides fine-grained access management to Azure resources.


#### Owner
* Full access to all resources
* Delegate access to others
* The Service Administrator and Co-Administrators are assigned the Owner role at the subscription scope
* Applies to all resource types.
  
#### Contributor
* Create and manage all of types of Azure resources
* Create a new tenant in Azure Active Directory
* Cannot grant access to others
* Applies to all resource types.
  
#### Reader	
* View Azure resources
* Applies to all resource types.

#### User Access Administrator	
* Manage user access to Azure resources


### Azure AD roles

Azure AD roles are used to manage Azure AD resources in a directory such as create or edit users, assign administrative roles to others, reset user passwords, manage user licenses, and manage domains.

#### Global Administrator	
* Manage access to all administrative features in Azure Active Directory, as well as services that federate to Azure Active Directory
* Assign administrator roles to others
* Reset the password for any user and all other administrators
* The person who signs up for the Azure Active Directory tenant becomes a Global Administrator.

#### User Administrator	
* Create and manage all aspects of users and groups
* Manage support tickets
* Monitor service health
* Change passwords for users, Helpdesk administrators, and other User Administrators
  
#### Billing Administrator	
* Make purchases
* Manage subscriptions
* Manage support tickets
* Monitors service health

### Azure roles VS Azure AD roles


Azure roles	| Azure AD roles
-----|------
Manage access to Azure resources	| Manage access to Azure Active Directory resources
Supports custom roles	|Supports custom roles
Scope can be specified at multiple levels (management group, subscription, resource group, resource)	| Scope is at the tenant level
Role information can be accessed in Azure portal, Azure CLI, Azure PowerShell, Azure Resource Manager templates, REST API	| Role information can be accessed in Azure admin portal, Microsoft 365 admin center, Microsoft Graph, AzureAD PowerShell


### Overlap

By default, Azure roles and Azure AD roles do not span Azure and Azure AD. However, if a Global Administrator elevates their access by choosing the Access management for Azure resources switch in the Azure portal, the Global Administrator will be granted the User Access Administrator role (an Azure role) on all subscriptions for a particular tenant.


<file_sep>+++
date = "2017-08-04T14:59:31+11:00"
title = "Monero Blockchain notes"
draft = true
+++

### Prerequisites

* You have a VPS or Dedicated Server. 
* You have some basic concept of blockchain, cryptocurrency. 
* You are happy to play round with infrastructures. 


### Windows

## Infrasture 
* Machine : VPS
* OS : Windows 2012 R2 64bit
* CPU: 
* Memory:
* Storage: 

 

## Install Msys2 

* Update package 

```bash
pacman -Syuu
```

* Install dependencies:

```bash
  pacman -S mingw-w64-x86_64-toolchain make mingw-w64-x86_64-cmake mingw-w64-x86_64-boost \
    mingw-w64-x86_64-openssl mingw-w64-x86_64-zeromq mingw-w64-x86_64-libsodium
```
* Build the  


### Linux <file_sep>+++
date = "2016-07-10T14:59:31+11:00"
title = "Build mobile app with web tech"
description="JavaScript, CSS, HTML are not just  web tech stacks, but also available for Mobile "
+++


## What is mobile app

> *A mobile application, basically, is a computer generated program designed and developed to run on iPhone, Android Smartphone, and many other mobile devices. In a nutshell, there are three types of apps*

> *__Native apps__ are specific to a given mobile platform (iOS or Android) using the development tools and language that the respective platform. Usaully it looks and performs the best.*

> *__HTML5 apps__ use standard web technologies—typically HTML5, JavaScript and CSS. This write-once-run-anywhere approach to mobile development creates cross-platform mobile applications that work on multiple devices.*

> *__Hybrid apps__ make it possible to embed HTML5 apps inside a thin native container, combining the good parts of __Native app__ and __HTML5 app__ elements.*

## Mobile app development

According above breif history we can image the mobile developer community has become asfragmented as the market. Mobile software developers work with different programming environments, different tools, and different programming languages.

After a few years of improvement, we can see some __Hybrid app__ based framework becomes more and more popular and shining. `ionic`, `nativescript` and `react native` are most promising frameworks which we should really look into. 

## Introduction of ionic developement

### Prerequisites

* Here we just introduce ionic 1.x. When I started investigating the `ionic`, the `ionic 2` just came out for a while. `ionic 2` study is on my todo list.
* You are familiar with web technologies, such as, HTML5, CSS, JavaScript, and you should have experience of nodejs and relevant skills.


### Getting started

* install ionic 1.x
* setup ionic and create new project ```demoApp```



```bash
ionic start demoApp slidemenu
cd demoApp
ionic platform add android 
ionic build android 
ionic emulate android 
```

### Install packages  

* Use `npm install` to install packages
* Folllowing is the `package.json`. You can tailor it on your own.

```json
{
  "name": "ionic-project",
  "version": "1.0.0",
  "description": "An Ionic project",
  "dependencies": {
    "gulp": "^3.5.6",
    "gulp-sass": "^2.0.4",
    "gulp-concat": "^2.2.0",
    "gulp-minify-css": "^0.3.0",
    "gulp-rename": "^1.2.0"
  },
  "devDependencies": {
    "bower": "^1.3.3",
    "gulp-util": "^2.2.14",
    "shelljs": "^0.3.0"
  },
  "cordovaPlugins": [
    "cordova-plugin-device",
    "cordova-plugin-console",
    "cordova-plugin-whitelist",
    "cordova-plugin-splashscreen",
    "cordova-plugin-statusbar",
    "ionic-plugin-keyboard"
  ],
  "cordovaPlatforms": []
}

```


### Debug

*  Browser is the best option for ionic mobile development debug tools
*  Use Telerik AppBuilder to debug


###  Test on emulator or device

*  Android 
  *  Download and install [Android SDK](https://developer.android.com/studio/index.html#downloads)
  *  Download install at least one sdk platform. 
    *  ionic only support Android 4.1.x or later, so you are better to install any sdk platform version 18+. 
    *  install x86 or x86_64 image for windows environment
    *  install Extra plugins: Google USB driver, X86 Emulator Accelerator
    *  Create AVD for your mobile app testing

*  iPhone
  *  Install AppBuilder on Visual Studio
  *  Install Genymotion 



### Troubleshooting

*  Android emulator accelerator error due to version is too low to support the system image
*   uninstall old version intel HAXM 
*   install new version manually from `<Android_SDK_Location>\extras\intel\Hardware_Accelerated_Execution_Manager`

*  If app is not working on emulator, check cordova plugins or manually install cordova plugins

    ```bash
    ionic plugins list
    ionic plugins add XXXXX
    ```
<file_sep>#!/bin/bash

echo "commit blog content to github"

git config user.name "<NAME>"
git config user.email "<EMAIL>"

# Add changes under folder content to git.
git add content/*
git add static/img/*

# Commit changes.
msg="commit blog content `date`"
# if  NOT "%1"=="" set msg=%1
if [ $# -eq 1 ] 
    then msg=$1
fi

git commit -m "$msg"

# Push source and build repos.
git push origin master<file_sep>+++
title="AWS - Cert Exams"
description="AWS Cert Exam Tips"
weight=9
draft=true
+++

## Solution Architect Associate

### Domains

### Tips

* If an instance in VPC is unable to communicate over certain protocol / port with another instance in the same VPC, then the problem is the security setting:
  
    - Security group / NACL of the instance and / or 
    - Security Group / NACL of the destination instance 
    - The problelm will never be routing table configuration, due to the default route entry
    - NACL are stateless. To allow certain traffic through it. Inbound and outbound rules both must be allowed. 

* Default VPC & Default Security Group

    - Have an inbound rule that allows all traffic sourced from the security group itself
    - Have all the outbound traffic allowed by default
    

### Scenario 1

- EC2-A and EC2-B both are in the same VPC
- EC2-A is protected by NACL-A and SecGrp-A
- EC2-B is protected by NACL-B and SecGrp-B
- EC2-A can ICMP ping EC2-B
- EC2-B can ICMP ping EC2-A

__If EC2-A can ping EC2-B, but EC2-B can NOT ping EC2-A, which component will cause this problem?__

- SecGrp-B outbound
- NACL-B outbound 
- NACL-A inbound 
- SecGrp-A inbound



## Solution Architect Professional




## Developer Associate


## Developer Professional




<file_sep>+++
date = "2017-01-11T11:59:31+11:00"
title = "JIRA Practices"
description="JIRA is a proprietary issue tracking product, developed by Atlassian. Today I recap the bullet points about how I work with client within JIRA "
draft="true"
+++

## What is JIRA

> Jira is a proprietary issue tracking product, developed by Atlassian. It provides bug tracking, issue tracking, and project management functions. Although normally styled JIRA, the product name is not an acronym, but a truncation of Gojira, the Japanese name for Godzilla,[6] itself a reference to Jira's main competitor, Bugzilla. It has been developed since 2002.


## How to cooperate with client via JIRA

> The instruction below is only for the client who has independent JIRA instance. Here I am not going to discuss how to cooperate with multiple clients and projects on the same JIRA instance. IMO, the strategy for that would be case by case. 

### Prerequisites

* Your client's business name is __ABC__. You help them to manage their JIRA. You are the administrator of JIRA instance. 

* You have created a project for client, named **ABC IT Project**

* You want to allow the client to access this project on JIRA, but you want to avoid client's unintented update to mess up the project management. 

* You want to cooperate with other project team which build some product communicating with your system

* You want to integrate with CI / CD tools


### Requirement of permission control

* Product owner, ScrumMaster, and development team  should be able to track the development progress and be aware of criticle issues, etc. 

* Business owners, executives, and managers as internal stakeholders should see the progress firsthand so that they can suggest course corrections. 

* Sales, marketing, support, legal, compliance, and other Scrum and non-Scrum development teams might want to attend sprint reviews to provide area-specific feedback, etc. 

* Customers, users, partners, and regulators as External stakeholders can provide valuable feedback to the Scrum team 


### Change default permission scheme

* Default permission scheme is the built-in scheme, which you cannot delete it, but you can change it.

* Choose `Isssues` > `Permission schemes `

* Change all permissions from `Project Access - Any logged in user` to `Project role - Administrator` except the following items.

    * View Development Tools
    * View Read-Only Workflow
    * Assignable User
    * Delete Own Comments
    * Edit Own Comments
    * Delete Own Attachments
    * Delete Own Worklogs
    * Edit Own Worklogs
    * Work On Issues

* Copy the default permission scheme and rename it to `<client_business_name> Permission Scheme`. In this case it would be `ABC Permission Scheme`. 

* Create a new group named **ABC Group**, 

* Update `ABC Permission Scheme` by granting `ABC Group` to some items.
    * Browse Projects
    * Create Issues
* Update the permission scheme of the project `ABC IT Project` from to `ABC Permission Scheme`. 

### Change user's group 

* Choose `User management` > `User` 
* Add the ABC's user to group `ABC Group`
* Remove the user from default `jira-software-user` and make sure the checkbox `Access JIRA Software` is selected.

 
### Time Tracking and Estimation

#### Enable Time Tracking

* Choose `Issues` > `Time Tracking` 
    * Enable the `Time Tracking` if it is inactive.

* Choose `Boards` > `View All Boards` 
    * Choose `Board Settings` of the board
    * Choose `Estimation`
    * Choose `Remaining Estimate and Time Spent` as Time Tracking


<file_sep>+++
title = "OS"
weight = 5
+++

{{%children style="card" description="true" sort="Weight" %}}<file_sep>+++
title = "IO, Json & XML"
description="Golang Introduction: IO, JSON, XML , Gob, Crypto "
weight=4
+++



### IO - Read & Write

#### Read from user input

```go
var (
    firstName, lastName, s string
    i                      int
    f                      float32
    input                  = "56.12 / 5212 / Go"
    format                 = "%f / %d / %s"
)

func main() {
    fmt.Println("Please enter your full name: ")
    fmt.Scanln(&firstName, &lastName)
    // fmt.Scanf("%s %s", &firstName, &lastName)
    fmt.Printf("Hi %s %s!\n", firstName, lastName) // Hi <NAME>
    fmt.Sscanf(input, format, &f, &i, &s)
    fmt.Println("From the string we read: ", f, i, s)
    // ouwtput: From the string we read: 56.12 5212 Go
}
```

#### Read command-line argument

```go
func main() {
    who := "Harry "
    if len(os.Args) > 1 {
        who += strings.Join(os.Args[1:], " ")
    }
    fmt.Println("Good Morning", who)
}
```


#### Read from file

* Read by lines

```go
func main() {
    inputFile, inputError := os.Open("hello.go")
    if inputError != nil {
        fmt.Printf("An error occurred on opening the inputfile\n" +
            "Does the file exist?\n" +
            "Have you got acces to it?\n")
        return // exit the function on error
    }
    defer inputFile.Close()  // Close file before exits the main func
    inputReader := bufio.NewReader(inputFile)
    for {
        inputString, readerError := inputReader.ReadString('\n')
        if readerError == io.EOF {
            return
        }
        fmt.Printf("The input was: %s", inputString)
    }
}
```

 ***NOTE: The End-of-line characters of text-files in Unix end on \n, but in Windows this is \r\n. By using the method ReadString or ReadBytes with \n as a delimiter you don’t have to worry about this.***

* Read entire file into a string


```go
func main() {
    inputFile := "hello.go"
    outputFile := "hello.go.txt"
    buf, err := ioutil.ReadFile(inputFile)
    if err != nil {
        fmt.Fprintf(os.Stderr, "File Error: %s\n", err)
        // panic(err.Error())
    }
    fmt.Printf("%s\n", string(buf))
    err = ioutil.WriteFile(outputFile, buf, 0x644)
    if err != nil {
        panic(err.Error())
    }
}
```


* Read into buffer

```go
for {
    n, err := inputReader.Read(buf)
    if (n == 0) { break}
}
```

* Read columns of data, such as csv

```go
func main() {
    file, err := os.Open("csv_data.txt")
    if err != nil {
        panic(err)
    }
    defer file.Close()
    var col1, col2, col3 []string
    for {
        var v1, v2, v3 string
        _, err := fmt.Fscanln(file, &v1, &v2, &v3)
        // scans until newline
        if err != nil {
            break
        }
        col1 = append(col1, v1)
        col2 = append(col2, v2)
        col3 = append(col3, v3)
    }
    fmt.Println(" Col 1 ",col1)
    fmt.Println(" Col 3 ",col2)
    fmt.Println(" Col 3 ",col3)
}
```

* Read from compressed file


```go
func main() {
    fName := "public.zip"
    var r *bufio.Reader
    fi, err := os.Open(fName)
    if err != nil {
        fmt.Fprintf(os.Stderr, "%v, Can’t open %s: error: %s\n", os.Args[0],
            fName, err)
        os.Exit(1)
    }
    fz, err := gzip.NewReader(fi)
    if err != nil {
        r = bufio.NewReader(fi)
    } else {
        r = bufio.NewReader(fz)
    }
    for {
        line, err := r.ReadString('\n')
        if err != nil {

            fmt.Println("Done reading file")
            os.Exit(0)
        }
        fmt.Println(line)
    }
}
```

#### Read with flag package

* The package flag has an extended functionality for parsing of command-line options.
* flag.PrintDefaults() prints out the usage information of the defined flag(s) 


```go
var NewLine = flag.Bool("n", false, "print on newline")

// echo -n flag, of type *bool
const (
    Space   = " "
    Newline = "\n"
)

func main() {
    flag.PrintDefaults()
    flag.Parse()
    var s string = ""
    for i := 0; i < flag.NArg(); i++ {
        if i > 0 {
            s += Space
        }
        s += flag.Arg(i)
    }
    if *NewLine { // -n is parsed, flag becomes true
        s += Newline
    }
    os.Stdout.WriteString(s)
}
// :: Output
// -------------------------
// command: go run program.go -n abc
// out : 
//   -n    print on newline
// abc
//----------------------------
// command: go run program.go -a abc
// flag provided but not defined: -a
// Usage of /tmp/go-build701354435/b001/exe/program:
//   -n    print on newline
// exit status 2
```

#### Read with flag parsing & buffer
* Assume there is a file named test.txt with some content
* Run Command: go build program.go && ./program test  

```go
func cat(r *bufio.Reader) {
    for {
        buf, err := r.ReadBytes('\n')
        if err == io.EOF {
            break
        }
        fmt.Fprintf(os.Stdout, "%s", buf)
    }
    return
}
func main() {
    flag.Parse()
    if flag.NArg() == 0 {
        cat(bufio.NewReader(os.Stdin))
    }

    for i := 0; i < flag.NArg(); i++ {
        f, err := os.Open(flag.Arg(i))
        if err != nil {
            fmt.Fprintf(os.Stderr, "%s:error reading from %s: %s\n",
                os.Args[0], flag.Arg(i), err.Error())
            continue
        }
        cat(bufio.NewReader(f))
    }
}
```

#### Read with flag parsing & slice
* Only different from previous sample is the cat function 

```go
func cat(f *os.File) {
    const NBUF = 512
    var buf [NBUF]byte
    for {
        switch nr, err := f.Read(buf[:]); true {
        case nr < 0:
            fmt.Fprintf(os.Stderr, "cat: error reading: %s\n", err.Error())
            os.Exit(1)
        case nr == 0: // EOF
            return
        case nr > 0:
            if nw, ew := os.Stdout.Write(buf[0:nr]); nw != nr {
                fmt.Fprintf(os.Stderr, "cat: error writing: %s\n",
                    ew)
            }
        }
    }
}

func main() {
    flag.Parse()
    if flag.NArg() == 0 {
        cat(os.Stdin)
    }

    for i := 0; i < flag.NArg(); i++ {
        f, err := os.Open(flag.Arg(i))
        if err != nil {
            fmt.Fprintf(os.Stderr, "%s:error reading from %s: %s\n",
                os.Args[0], flag.Arg(i), err.Error())
            os.Exit(1)
        }
        cat(f)
        f.Close()
    }
}
```

#### Write to a file

* Flags for open output file

    * os.O_RDONLY: the read flag for read-only access
    * os.WRONLY: the write flag for write-only access
    * os.O_CREATE : the create flag: create the file if it doesn’t exist
    * os.O_TRUNC : the truncate flag: truncate to size 0 if the file already exists


* Write with buffer

```go
func main() {
    outputFile, outputError := os.OpenFile("output.txt",
        os.O_WRONLY|os.O_CREATE, 0666)
    if outputError != nil {
        fmt.Printf("An error occurred with file creation\n")
        return
    }
    defer outputFile.Close()
    outputWriter := bufio.NewWriter(outputFile)
    outputString := "hello world!\n"
    for i := 0; i < 10; i++ {
        outputWriter.WriteString(outputString)
    }
    outputWriter.Flush()
}
```


* Write a file without buffer

    * Write the same content as previous sample

```go
func main() {
    os.Stdout.WriteString("hello, world\n")
    f, _ := os.OpenFile("output.txt", os.O_CREATE|os.O_WRONLY, 0)
    defer f.Close()
    for i := 0; i < 10; i++ {
        f.WriteString("hello world!\n")
    }
}
```

#### Write to standard output: interface

* The interface is fmt.Fprintf
* The fmt.Fprintf writes to a variable of type io.Writer.
* Any type that has a Write method, including os.Stdout, files (like os.File), pipes, network connections, channels, etc..., and also to write buffers from the bufio package.

```go
func main() {
    // unbuffered: os.Stdout implements io.Writer
    fmt.Fprintf(os.Stdout, "%s\n", "hello world! - unbuffered")
    // buffered:
    buf := bufio.NewWriter(os.Stdout)
    // and now so does buf:
    fmt.Fprintf(buf, "%s\n", "hello world! - buffered")
    buf.Flush()
}

```


#### Copy

* Simply use io.Copy method

```go
func main() {
    CopyFile("target_hello.txt", "hello.go")
    fmt.Println("Copy done!")

}
func CopyFile(dstName, srcName string) (written int64, err error) {
    src, err := os.Open(srcName)
    if err != nil {
        return
    }
    defer src.Close()
    dst, err := os.OpenFile(dstName, os.O_WRONLY|os.O_CREATE, 0644)
    if err != nil {
        return
    }
    defer dst.Close()
    return io.Copy(dst, src)
}
```

### JSON

* Mapping of Go type and Js type
    * Go: bool -> Js: boolean
    * Go: float64 -> Js: numbers
    * Go: string -> Js: strings
    * Go: nil -> Js: null

```go
type Address struct {
    Type    string
    City    string
    Country string
}
type VCard struct {
    FirstName string
    LastName  string
    Addresses []*Address
    Remark    string
}

func main() {
    pa := &Address{"private", "Aartselaar", "Belgium"}
    wa := &Address{"work", "Boom", "Belgium"}
    vc := VCard{"Jan", "Kersschot", []*Address{pa, wa}, "none"}
    // fmt.Printf("%v: \n", vc)
    // {<NAME> [0x126d2b80 0x126d2be0] none}:
    // JSON format:
    js, _ := json.Marshal(vc)
    fmt.Printf("JSON format: %s", js)
    // using an encoder:
    file, _ := os.OpenFile("vcard.json", os.O_CREATE|os.O_WRONLY, 0)
    defer file.Close()
    enc := json.NewEncoder(file)
    err := enc.Encode(vc)
    if err != nil {
        fmt.Println("Error in encoding json")
    }
}

// cat vcard.json | jq  # jq is a tool for json file
// ----------------------------------------------------
// {
//   "FirstName": "Jan",
//   "LastName": "Kersschot",
//   "Addresses": [
//     {
//       "Type": "private",
//       "City": "Aartselaar",
//       "Country": "Belgium"
//     },
//     {
//       "Type": "work",
//       "City": "Boom",
//       "Country": "Belgium"
//     }
//   ],
//   "Remark": "none"
// }
```

### XML 

* Sample xml 

```xml
<Person>
    <FirstName>Laura</FirstName>
    <LastName>Lynn</LastName>
</Person>
```
* The encoding/xml package also implements a simle xml parser (SAX) to read XML-data and parse it into its constituents.

```go

var t, token xml.Token
var err error

func main() {
    input :=
        "<Person><FirstName>Laura</FirstName><LastName>Lynn</LastName></Person>"
    inputReader := strings.NewReader(input)
    p := xml.NewDecoder(inputReader)
    for t, err = p.Token(); err == nil; t, err = p.Token() {
        switch token := t.(type) {
        case xml.StartElement:
            name := token.Name.Local
            fmt.Printf("Token name: %s\n", name)
            for _, attr := range token.Attr {
                attrName := attr.Name.Local
                attrValue := attr.Value
                fmt.Printf("An attribute is: %s %s\n", attrName,
                    attrValue)
                // ...
            }
        case xml.EndElement:
            fmt.Println("End of token")
        case xml.CharData:
            content := string([]byte(token))
            fmt.Printf("This is the content: %v\n", content)
        // ...
        default:
            // ...
        }
    }
}

//----------------------
// Token name: Person
// Token name: FirstName
// This is the content: Laura
// End of token
// Token name: LastName
// This is the content: Lynn
// End of token
// End of token
```

### Gob - Go binary format

* Simulate network communication

```go
type P struct {
    X, Y, Z int
    Name    string
}
type Q struct {
    X, Y *int32
    Name string
}

func main() {
    // Initialize the encoder and decoder. Normally enc and dec would
    // be bound to network connections and the encoder and decoder
    // would run in different processes.
    var network bytes.Buffer
    // Stand-in for a network connection
    enc := gob.NewEncoder(&network) // Will write to network.
    dec := gob.NewDecoder(&network)
    // Will read from network.
    // Encode (send) the value.
    err := enc.Encode(P{3, 4, 5, "Pythagoras"})
    if err != nil {
        log.Fatal("encode error:", err)
    }
    // Decode (receive) the value.
    var q Q
    err = dec.Decode(&q)
    if err != nil {
        log.Fatal("decode error:", err)
    }
    fmt.Printf("%q: {%d,%d}\n", q.Name, *q.X, *q.Y)
    // "Pythagoras": {3,4}
}
```

* Simulate file operation

```go
type Address struct {
    Type    string
    City    string
    Country string
}
type VCard struct {
    FirstName string
    LastName  string
    Addresses []*Address
    Remark    string
}

func main() {
    pa := &Address{"private", "Aartselaar", "Belgium"}
    wa := &Address{"work", "Boom", "Belgium"}
    vc := VCard{"Jan", "Kersschot", []*Address{pa, wa}, "none"}
    // fmt.Printf("%v: \n", vc)
    // {<NAME> [0x126d2b80 0x126d2be0] none}:
    // using an encoder:
    file, _ := os.OpenFile("vcard.gob", os.O_CREATE|os.O_WRONLY, 0)
    defer file.Close()
    enc := gob.NewEncoder(file)
    err := enc.Encode(vc)
    if err != nil {
        log.Println("Error in encoding gob")
    }
}
```

### Crypto

* The hash package: implements the adler32, crc32, crc64 and fnv checksums;
* The crypto package: implements other hashing algorithms like md4, md5, sha1, etc. and
complete encryption implementations for aes, blowfish, rc4, rsa, xtea, etc.

```go

func main() {
    hasher := sha1.New()
    io.WriteString(hasher, "test")
    b := []byte{}
    fmt.Printf("Result: %x\n", hasher.Sum(b))
    fmt.Printf("Result: %d\n", hasher.Sum(b))
    hasher.Reset()
    data := []byte("We shall overcome!")
    n, err := hasher.Write(data)
    if n != len(data) || err != nil {
        log.Printf("Hash write error: %v / %v", n, err)
    }
    checksum := hasher.Sum(b)
    fmt.Printf("Result: %x\n", checksum)
}

//----------------------------------
// Result: a94a8fe5ccb19ba61c4c0873d391e987982fbbd3
// Result: [169 74 143 229 204 177 155 166 28 76 8 115 211 145 233 135 152 47 187 211]
// Checksum: e2222bfc59850bbb00a722e764a555603bb59b2a
```
<file_sep>+++
title = "AWS: SQS,SNS,SES - 1"
description = "Introduction of SQS, SNS, SES"
weight=7
+++

## SQS


Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. SQS eliminates the complexity and overhead associated with managing and operating message oriented middleware, and empowers developers to focus on differentiating work. Using SQS, you can send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available. Get started with SQS in minutes using the AWS console, Command Line Interface or SDK of your choice, and three simple commands.



__Queue types__

### Standard Queues

* Unlimited Throughput: Standard queues support a nearly unlimited number of transactions per second (TPS) per API action.

* At-Least-Once Delivery: A message is delivered at least once, but occasionally more than one copy of a message is delivered.

* Best-Effort Ordering: Occasionally, messages might be delivered in an order different from which they were sent.

### FIFO Queues

* High Throughput: By default, FIFO queues support up to 300 messages per second (300 send, receive, or delete operations per second). When you batch 10 messages per operation (maximum), FIFO queues can support up to 3,000 messages per second. To request a quota increase, file a support request.

* Exactly-Once Processing: A message is delivered once and remains available until a consumer processes and deletes it. Duplicates aren't introduced into the queue.

* First-In-First-Out Delivery: The order in which messages are sent and received is strictly preserved (i.e. First-In-First-Out).


## SNS

Amazon Simple Notification Service (SNS) is a fully managed messaging service for both system-to-system and app-to-person (A2P) communication. It enables you to communicate between systems through publish/subscribe (pub/sub) patterns that enable messaging between decoupled microservice applications or to communicate directly to users via SMS, mobile push and email.

> There are two main uses for SNS. First, you can use it as a traditional pub/sub messaging system. An example here is a microservice architecture where Service A may be interested in updates to objects in Service B. Rather than Service B directly notifying Service A about the update, Service B can send a message to an SNS Topic with details about the update. Service A can subscribe to the topic and process the messages as they arrive. The main alternatives to using SNS in this manner are tools like RabbitMQ in pub/sub mode or NATS.

> The second core use case for SNS is to deliver messages to large numbers of end users, such as via mobile push notifications or SMS messages. SNS allows for extremely high fan-out in these use cases, as you can have up to 12.5 million subscribers to a single topic. This is great for blasting your users with updates about a new sale or news in your application.

## SES

Amazon Simple Email Service (SES) is a cost-effective, flexible, and scalable email service that enables developers to send mail from within any application. You can configure Amazon SES quickly to support several email use cases, including transactional, marketing, or mass email communications. Amazon SES's flexible IP deployment and email authentication options help drive higher deliverability and protect sender reputation, while sending analytics measure the impact of each email. With Amazon SES, you can send email securely, globally, and at scale.


> The core main use case for SES is sending email. There are two email types. First is transactional email. Transaction email is when you send an automated message to a specific person. For example, you could send a customer an email that their order has shipped, or you can alert a user about the subscription  ending.

> The second bucket of outgoing email is marketing email. This is when you send the same email to a large number of users at the same time. This could be notifying your email list of a large upcoming sale or making an announcement of a new product.<file_sep>+++
title = "AWS: EKS - 3"
description = "Cluster Autoscaler, Horizontal Pod Autoscaler, Vertical Pod Autoscaler"
weight=13
+++


## EKS - Part 3

### Cluster Autoscaler

The Kubernetes Cluster Autoscaler automatically adjusts the number of nodes in your cluster when pods fail to launch due to lack of resources or when nodes in the cluster are underutilized and their pods can be rescheduled onto other nodes in the cluster.

### Strategy of auto     scaling

* Stateful application 
  
If you are running a stateful application across multiple Availability Zones that is backed by Amazon EBS volumes and using the Kubernetes Cluster Autoscaler, you should configure multiple node groups, each scoped to a single Availability Zone.

* Other option

Create a single node group that spans multiple Availability Zones.


#### Single managed node group

Create an Amazon EKS cluster with a single managed node group

    eksctl create cluster --name pg-smng --version 1.15 --managed --asg-access


#### Node group per AZ

Create a cluster with a dedicated managed node group for each Availability Zone

    eksctl create cluster --name pg-ngaz --version 1.15 --without-nodegroup

For each Availability Zone in your cluster, use the following eksctl command to create a node group. 

    eksctl create nodegroup \
            --cluster pg-ngaz \
            --node-zones ap-southeast-2a \
            --name ap-southeast-2a \
            --asg-access \
            --node-type t3.medium \
            --nodes-min 1 --nodes 1 \
            --nodes-max 3 --managed


* Node Group IAM Policy

The Cluster Autoscaler requires the following IAM permissions to make calls to AWS APIs on your behalf. The tool __eksctl__ automatically provides and attaches to your worker node IAM roles, when it creates the node groups.

    {
        "Version": "2012-10-17",
        "Statement": [
                {
                "Action": [
                        "autoscaling:DescribeAutoScalingGroups",
                        "autoscaling:DescribeAutoScalingInstances",
                        "autoscaling:DescribeLaunchConfigurations",
                        "autoscaling:DescribeTags",
                        "autoscaling:SetDesiredCapacity",
                        "autoscaling:TerminateInstanceInAutoScalingGroup",
                        "ec2:DescribeLaunchTemplateVersions"
                ],
                "Resource": "*",
                "Effect": "Allow"
                }
        ]
    }


#### Deploy Autoscaler


Deploy the Cluster Autoscaler to your cluster with the following command.

    kubectl apply -f https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml

Add the cluster-autoscaler.kubernetes.io/safe-to-evict annotation to the deployment with the following command.

    kubectl -n kube-system annotate deployment.apps/cluster-autoscaler cluster-autoscaler.kubernetes.io/safe-to-evict="false"

Edit the Cluster Autoscaler deployment with the following command.

    kubectl -n kube-system edit deployment.apps/cluster-autoscaler  

Edit the cluster-autoscaler container command to replace <YOUR CLUSTER NAME> with your cluster's name, and add the following options.

    --balance-similar-node-groups
    --skip-nodes-with-system-pods=false

Final change will look like below

    spec:
      containers:
      - command:
        - ./cluster-autoscaler
        - --v=4
        - --stderrthreshold=info
        - --cloud-provider=aws
        - --skip-nodes-with-local-storage=false
        - --expander=least-waste
        - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/<YOUR CLUSTER NAME>
        - --balance-similar-node-groups
        - --skip-nodes-with-system-pods=false

Set the Cluster Autoscaler image tag

    kubectl -n kube-system set image deployment.apps/cluster-autoscaler cluster-autoscaler=asia.gcr.io/k8s-artifacts-prod/autoscaling/cluster-autoscaler:v1.15.6

Log Autoscaler

    kubectl -n kube-system logs -f deployment.apps/cluster-autoscaler


Add scale policy 

    


### Horizontal Pod Autoscaler


The Kubernetes Horizontal Pod Autoscaler automatically scales the number of pods in a deployment, replication controller, or replica set based on that resource's CPU utilization. This can help your applications scale out to meet increased demand or scale in when resources are not needed, thus freeing up your worker nodes for other applications. When you set a target CPU utilization percentage, the Horizontal Pod Autoscaler scales your application in or out to try to meet that target.


The Horizontal Pod Autoscaler is a standard API resource in Kubernetes that simply requires that a metrics source (such as the Kubernetes metrics server) is installed on your Amazon EKS cluster to work. 


Install metrics-server with __curl__ and __jq__

    DOWNLOAD_URL=$(curl -Ls "https://api.github.com/repos/kubernetes-sigs/metrics-server/releases/latest" | jq -r .tarball_url)
    DOWNLOAD_VERSION=$(grep -o '[^/v]*$' <<< $DOWNLOAD_URL)
    curl -Ls $DOWNLOAD_URL -o metrics-server-$DOWNLOAD_VERSION.tar.gz
    mkdir metrics-server-$DOWNLOAD_VERSION
    tar -xzf metrics-server-$DOWNLOAD_VERSION.tar.gz --directory metrics-server-$DOWNLOAD_VERSION --strip-components 1
    kubectl apply -f metrics-server-$DOWNLOAD_VERSION/deploy/1.8+/


#### Horizontal Autoscale Test

* Install httpd pod
  
        kubectl run httpd \
        --generator=run-pod/v1 \
        --image=httpd --requests=cpu=100m \
        --limits=cpu=200m --expose --port=80

* Run benchmark test
  
        kubectl run apache-bench \
            --generator=run-pod/v1 \
            -i --tty --rm --image=httpd \
            -- ab -n 900000 \
            -c 9999 http://httpd.default.svc.cluster.local/


### Vertical Pod Autoscaler

The Kubernetes Vertical Pod Autoscaler automatically adjusts the CPU and memory reservations for your pods to help "right size" your applications. This adjustment can improve cluster resource utilization and free up CPU and memory for other pods. This topic helps you to deploy the Vertical Pod Autoscaler to your cluster and verify that it is working.

### Deploy Vertical Autoscaler

Open a terminal window and navigate to a directory where you would like to download the Vertical Pod Autoscaler source code.

* Clone the kubernetes/autoscaler GitHub repository.

        git clone https://github.com/kubernetes/autoscaler.git

* Change to the vertical-pod-autoscaler directory.

        cd autoscaler/vertical-pod-autoscaler/

* (Optional) If you have already deployed another version of the Vertical Pod Autoscaler, remove it with the following command.

        ./hack/vpa-down.sh

* Deploy the Vertical Pod Autoscaler to your cluster with the following command.

        ./hack/vpa-up.sh

* Check Vertical Autoscaler pods

        kubectl get pods -n kube-system | grep vpa



#### Test Vertical Autoscaler 

* Deploy the hamster.yaml Vertical Pod Autoscaler example with the following command.

        kubectl apply -f examples/hamster.yaml

* Get the pods from the hamster example application.

        kubectl get pods -l app=hamster
        
        # Output:

        hamster-c7d89d6db-rglf5   1/1     Running   0          48s
        hamster-c7d89d6db-znvz5   1/1     Running   0          48s

* Describe one of the pods to view its CPU and memory reservation.

        kubectl describe pod hamster-c7d89d6db-rglf5

* Describe the hamster-vpa resource to view the new recommendation.

        kubectl describe vpa/hamster-vpa

        # Output
        Status:
        Conditions:
            Last Transition Time:  2020-02-11T13:31:48Z
            Status:                True
            Type:                  RecommendationProvided
        Recommendation:
            Container Recommendations:
            Container Name:  hamster
            Lower Bound:
                Cpu:     530m
                Memory:  262144k
            Target:
                Cpu:     587m
                Memory:  262144k
            Uncapped Target:
                Cpu:     587m
                Memory:  262144k
            Upper Bound:
                Cpu:     1
                Memory:  500Mi





<file_sep>+++
title = "F# Namespace, Module & Import"
description = "F# Namespace, Module & Open Declaration"
+++

## Namespace

> A namespace lets you organize code into areas of related functionality by enabling you to attach a name to a grouping of F# program elements. Namespaces are typically top-level elements in F# files.

Namespaces **cannot** directly contain values and functions. Instead, values and functions must be included in modules, and modules are included in namespaces. Namespaces can contain types, modules.

```fsharp
namespace [rec] [parent-namespaces.]identifier
```

## Modules

> A module is a grouping of F# code, such as values, types, and function values, in an F# program. Grouping code in modules helps keep related code together and helps avoid name conflicts in your program.

```fsharp
// Top-level module declaration.
module [accessibility-modifier] [qualified-namespace.]module-name
declarations
// Local module declaration.
module [accessibility-modifier] module-name =
    declarations
```

## OPen Declaration


> An import declaration specifies a module or namespace whose elements you can reference without using a fully qualified name.

```fsharp
open module-or-namespace-name
open type type-name
```

## Namespace Examples

- Example 1

```fsharp
namespace Widgets

type MyWidget1 =
    member this.WidgetName = "Widget1"

module WidgetsModule =
    let widgetName = "Widget2"

```

- Example 2

```fsharp
namespace Widgets

module WidgetModule1 =
   let widgetFunction x y =
      printfn "Module1 %A %A" x y
module WidgetModule2 =
   let widgetFunction x y =
      printfn "Module2 %A %A" x y

module useWidgets =

  do
     WidgetModule1.widgetFunction 10 20
     WidgetModule2.widgetFunction 5 6


```

- Example 3 - Recursive nampespaces

```fsharp
namespace rec MutualReferences

type Orientation = Up | Down
type PeelState = Peeled | Unpeeled

// This exception depends on the type below.
exception DontSqueezeTheBananaException of Banana

type Banana(orientation : Orientation) =
    member val IsPeeled = false with get, set
    member val Orientation = orientation with get, set
    member val Sides: PeelState list = [ Unpeeled; Unpeeled; Unpeeled; Unpeeled] with get, set

    member self.Peel() = BananaHelpers.peel self // Note the dependency on the BananaHelpers module.
    member self.SqueezeJuiceOut() = raise (DontSqueezeTheBananaException self) // This member depends on the exception above.

module BananaHelpers =
    let peel (b: Banana) =
        let flip (banana: Banana) =
            match banana.Orientation with
            | Up ->
                banana.Orientation <- Down
                banana
            | Down -> banana

        let peelSides (banana: Banana) =
            banana.Sides
            |> List.map (function
                         | Unpeeled -> Peeled
                         | Peeled -> Peeled)

        match b.Orientation with
        | Up ->   b |> flip |> peelSides
        | Down -> b |> peelSides
```

## Module Examples

- Example 1 - Recursive

```fsharp
namespace rec MutualReferences

type Orientation = Up | Down
type PeelState = Peeled | Unpeeled

// This exception depends on the type below.
exception DontSqueezeTheBananaException of Banana

type Banana(orientation : Orientation) =
    member val IsPeeled = false with get, set
    member val Orientation = orientation with get, set
    member val Sides: PeelState list = [ Unpeeled; Unpeeled; Unpeeled; Unpeeled] with get, set

    member self.Peel() = BananaHelpers.peel self // Note the dependency on the BananaHelpers module.
    member self.SqueezeJuiceOut() = raise (DontSqueezeTheBananaException self) // This member depends on the exception above.

module BananaHelpers =
    let peel (b: Banana) =
        let flip (banana: Banana) =
            match banana.Orientation with
            | Up ->
                banana.Orientation <- Down
                banana
            | Down -> banana

        let peelSides (banana: Banana) =
            banana.Sides
            |> List.map (function
                         | Unpeeled -> Peeled
                         | Peeled -> Peeled)

        match b.Orientation with
        | Up ->   b |> flip |> peelSides
        | Down -> b |> peelSides

```

## Open Examples

- Example 1

```fsharp
// Open a .NET Framework namespace.
open System.IO

// Now you do not have to include the full paths.
let writeToFile2 filename (text: string) =
  let stream1 = new FileStream(filename, FileMode.Create)
  let writer = new StreamWriter(stream1)
  writer.WriteLine(text)

writeToFile2 "file1.txt" "Testing..."

```
<file_sep>+++
title = "AWS: EKS - 6"
description = "Labs: GuestBook"
weight=16

draft=true      
+++


## EKS - Part 6

### Guestbook

* Create the Redis master replication controller.

        kubectl apply -f https://raw.githubusercontent.com/kubernetes/examples/master/guestbook-go/redis-master-controller.json


* Create the Redis master service.

        kubectl apply -f https://raw.githubusercontent.com/kubernetes/examples/master/guestbook-go/redis-master-service.json



* Create the Redis slave replication controller.

        kubectl apply -f https://raw.githubusercontent.com/kubernetes/examples/master/guestbook-go/redis-slave-controller.json


* Create the Redis slave service.

        kubectl apply -f https://raw.githubusercontent.com/kubernetes/examples/master/guestbook-go/redis-slave-service.json

* Create the guestbook replication controller.

        kubectl apply -f https://raw.githubusercontent.com/kubernetes/examples/master/guestbook-go/guestbook-controller.json


* Create the guestbook service.

        kubectl apply -f https://raw.githubusercontent.com/kubernetes/examples/master/guestbook-go/guestbook-service.json



* Query the services in your cluster and wait until the External IP column for the guestbook service is populated.

        kubectl get services -o wide

After your external IP address is available, point a web browser to that address at port 3000 to view your guest book. For example, 

        http://a2444a44644eb431ca8b9f7617d85aad-1238693525.ap-southeast-2.elb.amazonaws.com:3000/


<file_sep>+++
title = "Map, Function & Closure"
description="Golang Introduction: Map & Function & Closure "
weight=2
+++

### Map

- Maps are a special kind of data structure: an unordered collection of pairs of items, where one element of the pair is the key, and the other element, associated with the key, is the data or the value, hence they are also called associative arrays or dictionaries.

- The key type can be any type for which the operations == and != are defined, like string, int, float. The value type can be any type.

- Map is much faster than a linear search, but still around 100x slower than direct indexing in an array or slice; so if performance is very important try to solve the problem with slices.

#### Definition

Maps are reference types: memory is allocated with the make -function

* Initialization of a map: 
    
        var map1[keytype]valuetype = make(map[keytype]valuetype)

* or shorter with:

        map1 := make(map[keytype]valuetype)

* mapCreated is made in this way:

        mapCreated := make(map[string]float)

* which is equivalent to: 
    
        mapCreated := map[string]float{}

#### Does the key exist

- Delete the pair from the map

```
if _, ok := map1[key1]; ok {
    delete(map1, key1)
}
```

### Function

- Function overloading, that is coding two or more functions in a program with the same function name but a different parameter list and/or a different return-type(s), is not allowed in Go.

- The default way in Go is to pass a variable as an argument to a function by value: a copy is made of that variable (and the data in it).

- Named variables used as result parameters are automatically initialized to their zero-value, and once they receive their value, a simple (empty) return statement is sufficient; furthermore even when there is only 1 named return variable, it has to be put inside ( )

#### Defer

- The defer keyword allows us to postpone the execution of a statement or a function until the end of the enclosing (calling) function: it executes something (a function or an expression) when the enclosing function returns (after every return and even when an error occurred in the midst of executing the function, not only a return at the end of the function), but before the }

- When many defer’s are issued in the code, they are executed at the end of the function in the inverse order (like a stack or LIFO): the last defer is first executed, and so on.

- Sample

    ```go
    func f() {
        for i := 0; i < 5; i++ {
            defer fmt.Printf(“%d “, i)
        }
    }
    // output : 4 3 2 1 0
    ```

- Defer allows us to guarantee that certain clean-up tasks are performed before we return from a function.

#### Recursive

* A function that call itself in its body is called recursive.
* An important problem when using recursive functions is stack overflow: this can occur when a large number of recursive calls are needed and the programs runs out of allocated stack memory. This can be solved by using a technique called lazy evaluation, implemented in Go with a channel and a goroutine.


    ```go
    package main
    import "fmt"

    func main() {
        result := 0
        for i:=0; i <= 10; i++ {
            result = fibonacci(i)
            fmt.Printf("fibonacci(%d) is: %d\n", i, result)
        }
    }

    func fibonacci(n int) (res int) {
        if n <= 1 {
            res = 1
        } else {
            res = fibonacci(n-1) + fibonacci(n-2)
        }
        return
    }
    ```

#### Callback

* Functions can be used as parameters in another function, the passed function can then be called within the body of that function, that is why it is commonly called a callback.


```go
func main() {
    callback(1, Add)
}
func Add(a, b int) {
    fmt.Printf("The sum of %d and %d is: %d\n", a, b, a+b)
}
func callback(y int, f func(int, int)) {
    f(y, 2)
    // this becomes Add(1, 2)
}
```

#### Closures (function literals)

* Sample 

```go
plus := func(x, y int) int { return x + y }
plus( 1,2) // 3

// invoke func immediatley
func(x, y int) int { return x + y }( 1, 2) // 3
```


#### Closures - return another function

* Use return function for Debugging

    ```go
    where := func() {
        _, file, line, _ := runtime.Caller(1)
        log.Printf(“%s:%d”, file, line)
    }

    func Func () {
        //....do sth 
        where ()
        // ....do another thing
    }
    ```


<file_sep>+++
title = "Blogs"
weight = 5
+++

{{%children style="card" description="true" %}}<file_sep>+++


date = "2018-01-06T14:59:31+11:00"
title = "Raspberry Pi setup"
description="How to setup Raspberry Pi as file server"
draft = false
+++

Prelude

> *This note is mainly to record how to setup Raspberry Pi as file server.


## Prerequisites

* You have a Raspberry Pi with pre-installed raspbian SD card
* You are happy to get your hands dirty
* You have some basic computer concept.
* RPi is short for Raspberry Pi  

## My Raspberry Pi is a bit old

I only have the RPi 1 model B with pre-installed raspbian SD card in place. It is quite outdated. If you don’t know the model of your RPi, please don’t worry it now. I will explain how to get the info later. I got this RPi as a gift 2 years ago. I left it in the garage and totally forgot it, until I cleaned up my garage a couple months ago. Actually I loved the old model with transparent plastic box more than the new one. I knew if I continued to leave it in the garage, it would be a rubbish soon, because it is not easy to find some equipment or software compatible with the old RPi. Luckily the lifespan of RPi is much longer than the mobile phone, but it still took me some effort to setup the wifi adapter.

After 6-hour on and off, I got it up and run. Honestly I’m not a hardware guru, but I’m so happy not to throw this beautiful (my aesthetics is sort of quirky) box into the bin. I captured the home screen of Kodi, the media center and mobile control app.

Home page of kodai

![KodiTVHomePage](/img/kodi-tv-homepage.jpg)

Screenshot of kodi mobile app on my android phone.

![kodi-mobile-app](/img/kodi-mobile-app.png)

## How to start
There is no wifi or bluetooth support on this model. I have to connect this tiny box to my switch via cable all the time. There is a small problem, because my switch is far away from my laptop, monitor, keyboard, etc. and I don’t have a cable long enough to connect the RPi and switch.

First thing first, I need to setup ssh server, and change the configuration to allow password login, also make it auto-start after reboot. To do so I just need monitor and keyboard. 

### Connect the RPi with monitor and keyboard

* Reset pasword of `pi`

```bash
sudo passwd pi
```

### SSH server setup

```bash
sudo apt-get install openssh-server 

### backup default config 
sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.ori
sudo chmod a-w /etc/ssh/sshd_config.ori

### use any editor to update sshd_config 
sudo nano /etc/ssh/sshd_config 

### uncomment  PasswordAuthentication yes to allow remote password login

### setup ssh auto-start onboot
sudo update-rc.d ssh defaults

### reboot
sudo reboot

### Check the ssh is running after reboot
sudo service ssh status 

### You should see sth as below
[ok] sshd is running

### Turn off Pi 
sudo poweroff
```

### Connect RPi with the switch

After all above is done, you can disconnect the monitor and keyboard, and connect the RPi with the switch (or modem). Once the power is on, you should be able to access the RPi from you PC or laptop. 


### Find the ip address

Access the admin home page of my switch via browser. e.g. `http://192.168.0.1/index.html` (The actual URL depends on your switch or modem. You can find it on the label sticked on the back or bottom.)

If you forget your password to login the admin page, you still can reset your swtich. If your modem is 3 in 1 model including switch, you need to make sure you have the `ID and Password` to access the internet before you reset it. 

After you login successfully, you just need to expand main menus find a menu called `DHCP`. e.g.

```
Basic Setup
|__ ...
Advanced Setup
|__ ...
Device
|__ DHCP
|__ WAN
|__ ...

```

You will see table as blew.


 Hostname | MAC Address | IP Address |    Expires In 
-----|---|----|---
 PC-1 | 2f:3f:09:ff:f5:24 | 192.168.1.7 | x hours x mins 
 PC-2 | c0:9f:05:ff:f9:14 | 192.168.1.8 | x hours x mins 
 Laptop-1 | b0:f6:05:e2:f5:99 | 192.168.1.9 | x hours x mins 
 raspberrypi| a5:06:b2:07:c4:03 | 192.168.1.10 | x hours x mins 


### Access RPi with your laptop

* From Linux or Mac 

```
ssh username@192.168.1.10

### type yes 
### type the password 

```

* For widnows

You need to download a ssh tool. If you installed git before, you would have it on your computer. Otherwise, you need to install a SSH too. I recommend you to install [Putty](http://www.putty.org/). It is free and quite handy.

After you install and launch Putty, you just need to type in the IP address `192.168.1.10` to the field `Host Name (or IP Address)`, then click button `Open`. 

```
### type in pi as login user
login as: pi

### type in password
pi@1192.168.3.11's password: 

```


## Access RPi via VNC

### Setup VNC server on RPi 

```bash
sudo apt-get update
sudo apt-get install tightvncserver
```

* Launch VNC server and setup pasword

```bash
/usr/bin/tightvncserver

### Setup password for remote access. 
### View only password is not necessary

### setup VNC server to auto start
sudo update_rc.d tightvncserver defaults

sudo reboot
```



### Setup VNC client on your PC

Linux: Use __xRDP__
I believe you can figure it out yourself, if you used Linux as desktop. 

Windows: Install [RealVNC Viewer](https://www.realvnc.com/en/connect/download/viewer/) as VNC client 

* Laucn the VNC Viewer and create a new connection 

![Screen-VNC-new-connection](/img/raspberry-pi-vnc-new-client.jpg)


* Type in the VNC password and you can login RPi with GUI

![Screenshot-VNC](/img/raspberry-pi-vnc.jpg)



After all above is done, you have your RPi ready. You can choose what you want to build on it. Considering its CPU and RAM, it is not sufficient to be used as daily desktop PC, but it is still enough to work as a server. e.g. File Server, Web Server, Email Server, FTP server or Media Center.

Now I want to make a file server and media center on it.


### Setup File Server via Samba

* Attach external storage to your RPi. The capacity of preinstalled SD card has only 8G space, so I attached my portal hard drives to RPi. You can attach the PC hard drive, USD or another SD card via adapter. It is really up to what you have in place. 

> I want to make one as public share folder without authentication, and the other needs password to access. 


* If your hard drive or USB is `ntfs`, the RPi might not recognize your device. You can simply install a package to make it work.

```bash
sudo apt-get install ntfs-3g
```

* Get drive info after attach two hard drives

```bash
sudo lsblk

### You will see the tree structure of drives
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0   1.8T  0 disk
└─sda1        8:1    0   1.8T  0 part /media/mydrive1
└─sda2        8:2    0   870G  0 part /media/mydrive2
mmcblk0     179:0    0   7.4G  0 disk
├─mmcblk0p1 179:1    0    56M  0 part /boot
└─mmcblk0p2 179:2    0   7.4G  0 part /
```


* Remount the drives with proper name

```bash
sudo su ### switch to root
cd /media
umount mydrive1 
umount mydrive2

mkdir public private 
mount -o rw /sda/sda1 public
mount -o rw /sda/sda2 private
```

* Change `fstab` to support read and write permission

```bash 
sudo nano /etc/fstab
```

* Add following lines to the end of file. The format type of my drives are `ntfs`. If you are not sure what file system type is, you can run this command to check `sudo lsblk -o name,fstype`

```
/dev/sda1  /media/public       ntfs   nofail,noatime    0    0
/dev/sda2  /media/private       ntfs   nofail,noatime    0    0
```

* After you complete above changes, you will see the difference by typing the command `sudo lsblk`

```
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0   1.8T  0 disk
└─sda1        8:1    0   1.8T  0 part /media/public
└─sda2        8:2    0   870G  0 part /media/private
mmcblk0     179:0    0   7.4G  0 disk
├─mmcblk0p1 179:1    0    56M  0 part /boot
└─mmcblk0p2 179:2    0   7.4G  0 part /
```

* Install Samba 

```bash
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install samba samba-common-bin
```

* Setup Samba configuration

## Backup original config and update the config

```bash
sudo su
cd /etc/samba
cp smb.conf smb.conf.ori

nano smb.conf
```

### Change the line below: 
    * `wins support = no` to `wins support = yes`

### Add follow lines to end of the file

```
[public]
comment = Share Folder
path = /media/public
create mask = 0665
directory mask = 0775
read only = no
guest ok = yes

[private]
comment = Private Folder
path = /media/private
valid users = root,smbu
force user = smbu
create mask = 0777
directory mask = 0777
writable = yes
browsable = yes
read only = no
guest ok = yes
```

### Add new user `smbu` for remote access. In case you 

```bash
sudo useradd smbu
sudo passwd smbu
sudo usermod -a -G root smbu
sudo smbpasswd smbu ## setup pasword for remote access
```

## Access the network folder

### Linx 

I have no any problem to access the both netowrk drives via Linux. 

### Windows

It took me some time to make it work for me. There are some bullet points, which may help you for trouble shooting.

* Please use `WORKGROUP` instead of domain. 
* Please keep name of workgroup as `WORKGROUP`
* Turn on the network discovery     
`Control Panel > All Control Panel Items> Network and Sharing Cente > Advanced sharing settings`
* Reboot the PC or laptop



## Get accurate version of RPi model 

* Get the revision code 

```bash
cat /etc/cpuinfo
```

* Check the table below to find your model

MODEL AND PI REVISION |    MEMORY |    HARDWARE REVISION CODE FROM CPUINFO
---|-----|------
Model B Revision 1.0    | 256MB |     0002
Model B Revision 1.0 + ECN0001 (no fuses, D14 removed)    | 256MB |     0003
Model B Revision 2.0 Mounting holes    | 256MB |     0004 0005 0006
Model A Mounting holes    | 256MB |     0007,0008,0009
Model B Revision 2.0 Mounting holes    | 512MB |     000d 000e 000f
Model B+    | 512MB |     0010
Compute Module    | 512MB |     0011
Model A+    | 256MB |     0012
Pi 2 Model B    | 1GB |     a01041 (Sony, UK) a21041 (Embest, China)
PiZero    | 512MB |     900092(no camera connector) 900093(camera connector)
Pi 3 Model B    | 1GB |     a02082 (Sony, UK) a22082 (Embest, China)
PiZero W    | 512MB |     9000c1

## Setup Wifi Adapter

Wifi adapter is not necessary for media centre, but it would save some effort to move your RPi around in your place, especially you want to connect your RPi with different devices from time to time.

I bought a D-Link adapter, which is dwa-131 with usb 2.0. This is the oldest one I can find in the store. If you are going to buy wifi adapter for old Unix-like system, please don’t buy the latest model. You will find you are trapped into incompatible issue between wifi drive and Linux kernel. You may have to upgrade the kernel or rebuild the drive.

As you know, there is always some hiccup to find the correct wifi drive to support your portable wifi adapter. It took me a while to find the proper way to install the wifi adapter drive. If you have the RPi 2⁄3, it would be much more easier. My RPi 1 model B comes with kernel 4.1.18*. I cannot find the source code of wifi drive which supports this old kernel today, and I don’t want to upgrade and rebuild the kernel.

Finally, I found a post on RPi's forum which solved my problem. Link of [MrEngman's post](https://www.raspberrypi.org/forums/viewtopic.php?f=45&t=103989&p=1048709&hilit=Realtek+RTL8192EU+ID+0BDA%3A818B+WiFi+drivers+for+Raspbian#p1048709). He updated on April aobut the dropbox issue and alternative solution. 

Download and install the new version of the script with commands

```
sudo wget http://www.fars-robotics.net/install-wifi -O /usr/bin/install-wifi
sudo chmod +x /usr/bin/install-wifi

### Shows details on using it.
sudo install-wifi -h


### To install the driver on your current kernel you should just need to run command
sudo install-wifi

### Check the wifi interface after installation
ifconfig -a
```

Setup Wifi password. You can simply do it via GUI application or via command lines if you like. Please check out the official document as below.

https://www.raspberrypi.org/documentation/configuration/wireless/




## Install Kodi as media centre

If you have NOOBS in the place, then you have everything you need. Because I don't have it, I follow the official instruction to install kodi. It is a simple way to convert your RPi into a media centre without scratching your head too much. 

```bash

### Install kodi
sudo apt-get update
sudo apt-get install kodi


### Config kodi
sudo nano /etc/default/kodi
ENABLED=1

```

Reboot the RPi, before you reboot it please make sure your TV’s HDMI has plugged into RPi. After a couple minutes, you will see the home page as I posted above. Don’t forget to install remote control app on your mobile. I pretty sure you find some remote control app for Kodi on your phone. Finally, you can enjoy your home media center.

<file_sep>+++
title = "F# Collections 1"
description = "F# Array & Slicing"
weight = 5
+++


## Array

Arrays are fixed-size, zero-based, mutable collections of consecutive data elements that are all of the same type.

### Create array

```fsharp
// 
let array1 = [| 1; 2; 3 |]

// Put each element on a separate line, in which case the semicolon separator is optional.
let array1 =
    [|
        1
        2
        3
     |]

// The type of the array elements is inferred from the literals used and must be consistent. 
//  The following code causes an error because 1.0 is a float and 2 and 3 are integers.
// Causes an error.
// let array2 = [| 1.0; 2; 3 |]


// Use sequence expressions to create arrays.
let array3 = [| for i in 1 .. 10 -> i * i |]

// use Array.zeroCreate
let arrayOfTenZeroes : int array = Array.zeroCreate 10

// use Array.empty
let myEmptyArray = Array.empty
printfn "Length of empty array: %d" myEmptyArray.Length
// Length of empty array: 0

// use Array.create
printfn "Array of floats set to 5.0: %A" (Array.create 10 5.0)
//  Area of floats set to 5.0: [|5.0; 5.0; 5.0; 5.0; 5.0; 5.0; 5.0; 5.0; 5.0; 5.0|]

// Array.init
printfn "Array of squares: %A" (Array.init 10 (fun index -> index * index))
// Array of squares: [|0; 1; 4; 9; 16; 25; 36; 49; 64; 81|]
```
### Copy

```fsharp
open System.Text

let firstArray : StringBuilder array = Array.init 3 (fun index -> new StringBuilder(""))
let secondArray = Array.copy firstArray
// Reset an element of the first array to a new value.
firstArray[0] <- new StringBuilder("Test1")
// Change an element of the first array.
firstArray[1].Insert(0, "Test2") |> ignore
printfn "%A" firstArray
printfn "%A" secondArray
// output
// [|Test1; Test2; |]
// [|; Test2; |]
```

### Access

```fsharp 
// Accesses elements from 0 to 2.
array1[0..2]

// Accesses elements from the beginning of the array to 2.
array1[..2]

// Accesses elements from 2 to the end of the array.
array1[2..]
```

### Functions

```fsharp
// Array.sub 
let a1 = [| 0 .. 99 |]
let a2 = Array.sub a1 5 10
printfn "%A" a2
// output
// [|5; 6; 7; 8; 9; 10; 11; 12; 13; 14|]


// Array.append creates a new array by combining two existing arrays.
printfn "%A" (Array.append [| 1; 2; 3|] [| 4; 5; 6|])
// output
// [|1; 2; 3; 4; 5; 6|]




// Array.choose selects elements of an array to include in a new array. 
printfn "%A" (Array.choose (fun elem -> if elem % 2 = 0 then
                                            Some(float (elem*elem - 1))
                                        else
                                            None) [| 1 .. 10 |])
// output
// [|3.0; 15.0; 35.0; 63.0; 99.0|]


// Array.collect runs a specified function on each array element of an existing array and then collects the elements generated by the function and combines them into a new array. 
printfn "%A" (Array.collect (fun elem -> [| 0 .. elem |]) [| 1; 5; 10|])
// output
// [|0; 1; 0; 1; 2; 3; 4; 5; 0; 1; 2; 3; 4; 5; 6; 7; 8; 9; 10|]


Array.concat takes a sequence of arrays and combines them into a single array. The following code demonstrates Array.concat.

F#

Copy
Array.concat [ [|0..3|] ; [|4|] ]
//output [|0; 1; 2; 3; 4|]

Array.concat [| [|0..3|] ; [|4|] |]
//output [|0; 1; 2; 3; 4|]



// Array.filter takes a Boolean condition function and generates a new array that contains only those elements from the input array for which the condition is true.
printfn "%A" (Array.filter (fun elem -> elem % 2 = 0) [| 1 .. 10|])
// The output
// [|2; 4; 6; 8; 10|]


// Array.rev generates a new array by reversing the order of an existing array. 
let stringReverse (s: string) =
    System.String(Array.rev (s.ToCharArray()))

printfn "%A" (stringReverse("!dlrow olleH"))
// The output 
// "Hello world!"

// You can easily combine functions in the array module that transform arrays by using the pipeline operator (|>)
[| 1 .. 10 |]
|> Array.filter (fun elem -> elem % 2 = 0)
|> Array.choose (fun elem -> if (elem <> 8) then Some(elem*elem) else None)
|> Array.rev
|> printfn "%A"
// output
// [|100; 36; 16; 4|]
```
### Multidimensional arrays



```fsharp
let my2DArray = array2D [ [ 1; 0]; [0; 1] ]
let arrayOfArrays = [| [| 1.0; 0.0 |]; [|0.0; 1.0 |] |]
let twoDimensionalArray = Array2D.init 2 2 (fun i j -> arrayOfArrays[i][j])
```

### Slicing

```fsharp
// Get rows 1 to N from an NxM matrix (returns a matrix):
matrix[1.., *]

// Get rows 1 to 3 from a matrix (returns a matrix):
matrix[1..3, *]

// Get columns 1 to 3 from a matrix (returns a matrix):
matrix[*, 1..3]

// Get a 3x3 submatrix:
matrix[1..3, 1..3]

// Get row 3 from a matrix as a vector:
matrix[3, *]

// Get column 3 from a matrix as a vector:
matrix[*, 3]
```


### Matrix & slicing

```fsharp 

type Matrix<'T>(N: int, M: int) =
    let internalArray = Array2D.zeroCreate<'T> N M

    member this.Item
        with get(a: int, b: int) = internalArray[a, b]
        and set(a: int, b: int) (value:'T) = internalArray[a, b] <- value

    member this.GetSlice(rowStart: int option, rowFinish : int option, colStart: int option, colFinish : int option) =
        let rowStart =
            match rowStart with
            | Some(v) -> v
            | None -> 0
        let rowFinish =
            match rowFinish with
            | Some(v) -> v
            | None -> internalArray.GetLength(0) - 1
        let colStart =
            match colStart with
            | Some(v) -> v
            | None -> 0
        let colFinish =
            match colFinish with
            | Some(v) -> v
            | None -> internalArray.GetLength(1) - 1
        internalArray[rowStart..rowFinish, colStart..colFinish]

    member this.GetSlice(row: int, colStart: int option, colFinish: int option) =
        let colStart =
            match colStart with
            | Some(v) -> v
            | None -> 0
        let colFinish =
            match colFinish with
            | Some(v) -> v
            | None -> internalArray.GetLength(1) - 1
        internalArray[row, colStart..colFinish]

    member this.GetSlice(rowStart: int option, rowFinish: int option, col: int) =
        let rowStart =
            match rowStart with
            | Some(v) -> v
            | None -> 0
        let rowFinish =
            match rowFinish with
            | Some(v) -> v
            | None -> internalArray.GetLength(0) - 1
        internalArray[rowStart..rowFinish, col]

module test =
    let generateTestMatrix x y =
        let matrix = new Matrix<float>(3, 3)
        for i in 0..2 do
            for j in 0..2 do
                matrix[i, j] <- float(i) * x - float(j) * y
        matrix

    let test1 = generateTestMatrix 2.3 1.1
    let submatrix = test1[0..1, 0..1]
    printfn $"{submatrix}"

    let firstRow = test1[0,*]
    let secondRow = test1[1,*]
    let firstCol = test1[*,0]
    printfn $"{firstCol}"
```

### Search


```fsharp 

// Array.find takes a Boolean function and returns the first element for which 
// the function returns true, or raises a 
// System.Collections.Generic.KeyNotFoundException if no element that satisfies
// the conditionis found. 
// Array.findIndex is like Array.find, except that it returns the index of 
// the element instead of the element itself.

// The following code uses Array.find and Array.findIndex to locate a number that 
// is both a perfect square and perfect cube.

let arrayA = [| 2 .. 100 |]
let delta = 1.0e-10
let isPerfectSquare (x:int) =
    let y = sqrt (float x)
    abs(y - round y) < delta
let isPerfectCube (x:int) =
    let y = System.Math.Pow(float x, 1.0/3.0)
    abs(y - round y) < delta
let element = Array.find (fun elem -> isPerfectSquare elem && isPerfectCube elem) arrayA
let index = Array.findIndex (fun elem -> isPerfectSquare elem && isPerfectCube elem) arrayA
printfn "The first element that is both a square and a cube is %d and its index is %d." element index
// The output is as follows.
// The first element that is both a square and a cube is 64 and its index is 62.

// ----------------------------------------------------------------------------
// Array.tryFind is like Array.find, except that its result is an option type, 
// and it returns None if no element is found. Array.tryFind should be used 
// instead of Array.find when you do not know whether a matching element is 
// in the array. Similarly, Array.tryFindIndex is like Array.findIndex except 
// that the option type is the return value. If no element is found, the option is None.

let delta = 1.0e-10
let isPerfectSquare (x:int) =
    let y = sqrt (float x)
    abs(y - round y) < delta
let isPerfectCube (x:int) =
    let y = System.Math.Pow(float x, 1.0/3.0)
    abs(y - round y) < delta
let lookForCubeAndSquare array1 =
    let result = Array.tryFind (fun elem -> isPerfectSquare elem && isPerfectCube elem) array1
    match result with
    | Some x -> printfn "Found an element: %d" x
    | None -> printfn "Failed to find a matching element."

lookForCubeAndSquare [| 1 .. 10 |]
lookForCubeAndSquare [| 100 .. 1000 |]
lookForCubeAndSquare [| 2 .. 50 |]
// The output is as follows.
// Found an element: 1
// Found an element: 729
// Failed to find a matching element.


// ----------------------------------------------------------------------------
// Use Array.tryPick when you need to transform an element in addition to finding it. 
// The result is the first element for which the function returns the transformed 
// element as an option value, or None if no such element is found.
// The following code shows the use of Array.tryPick. In this case, instead of
// a lambda expression, several local helper functions are defined to simplify the code.

let findPerfectSquareAndCube array1 =
    let delta = 1.0e-10
    let isPerfectSquare (x:int) =
        let y = sqrt (float x)
        abs(y - round y) < delta
    let isPerfectCube (x:int) =
        let y = System.Math.Pow(float x, 1.0/3.0)
        abs(y - round y) < delta
    // intFunction : (float -> float) -> int -> int
    // Allows the use of a floating point function with integers.
    let intFunction function1 number = int (round (function1 (float number)))
    let cubeRoot x = System.Math.Pow(x, 1.0/3.0)
    // testElement: int -> (int * int * int) option
    // Test an element to see whether it is a perfect square and a perfect
    // cube, and, if so, return the element, square root, and cube root
    // as an option value. Otherwise, return None.
    let testElement elem =
        if isPerfectSquare elem && isPerfectCube elem then
            Some(elem, intFunction sqrt elem, intFunction cubeRoot elem)
        else None
    match Array.tryPick testElement array1 with
    | Some (n, sqrt, cuberoot) -> printfn "Found an element %d with square root %d and cube root %d." n sqrt cuberoot
    | None -> printfn "Did not find an element that is both a perfect square and a perfect cube."

findPerfectSquareAndCube [| 1 .. 10 |]
findPerfectSquareAndCube [| 2 .. 100 |]
findPerfectSquareAndCube [| 100 .. 1000 |]
findPerfectSquareAndCube [| 1000 .. 10000 |]
findPerfectSquareAndCube [| 2 .. 50 |]
// The output is as follows.
// Found an element 1 with square root 1 and cube root 1.
// Found an element 64 with square root 8 and cube root 4.
// Found an element 729 with square root 27 and cube root 9.
// Found an element 4096 with square root 64 and cube root 16.
// Did not find an element that is both a perfect square and a perfect cube.
```

### Modify arrays

Array.set sets an element to a specified value. Array.fill sets a range of elements in an array to a specified value. The following code provides an example of Array.fill.

```fsharp
let arrayFill1 = [| 1 .. 25 |]
Array.fill arrayFill1 2 20 0
printfn "%A" arrayFill1

// output
// [|1; 2; 0; 0; 0; 0; 0; 0; 0; 0; 0; 0; 0; 0; 0; 0; 0; 0; 0; 0; 0; 0; 23; 24; 25|]
```

### Convert to and from other types

Array.ofList creates an array from a list. Array.ofSeq creates an array from a sequence. Array.toList and Array.toSeq convert to these other collection types from the array type.
<file_sep>+++
title = "AWS: EKS - 5"
description = "Monitoring, Analytics"
weight=15
+++


## EKS - Part 5

### Metrics Server

The Kubernetes metrics server is an aggregator of resource usage data in your cluster, and it is not deployed by default in Amazon EKS clusters. The metrics server is commonly used by other Kubernetes add ons, such as the Horizontal Pod Autoscaler or the Kubernetes Dashboard. 

* Deploy the metrics server

        kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml


* Verify that the metrics-server deployment

        kubectl get deployment metrics-server -n kube-system



### Prometheus


The Kubernetes API server exposes a number of metrics that are useful for monitoring and analysis. These metrics are exposed internally through a metrics endpoint that refers to the /metrics HTTP API. Like other endpoints, this endpoint is exposed on the Amazon EKS control plane. 

Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. Since its inception in 2012, many companies and organizations have adopted Prometheus, and the project has a very active developer and user community. It is now a standalone open source project and maintained independently of any company. To emphasize this, and to clarify the project's governance structure, Prometheus joined the Cloud Native Computing Foundation in 2016 as the second hosted project, after Kubernetes.


#### Deploying Prometheus


* Create a Prometheus namespace.

        kubectl create namespace prometheus

* Install __helm__

        ## MAC
        brew install helm

        ## Linux
        curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 > get_helm.sh
        chmod 700 get_helm.sh
        ./get_helm.sh

        ## Add stable repo to helm
        helm repo add stable https://kubernetes-charts.storage.googleapis.com/


* Deploy Prometheus.

        helm install prometheus stable/prometheus \
                --namespace prometheus \
                --set alertmanager.persistentVolume.storageClass="gp2",server.persistentVolume.storageClass="gp2"

* Verify that all of the pods in the prometheus namespace are in the READY state.

        kubectl get pods -n prometheus

### Grafana

Set the storage class to gp2, admin password, configuring the datasource to point to Prometheus and creating an external load balancer for the service.

    kubectl create namespace grafana
    helm install grafana stable/grafana \
            --namespace grafana \
            --set persistence.storageClassName="gp2" \
            --set adminPassword='<PASSWORD>' \
            --set datasources."datasources\.yaml".apiVersion=1 \
            --set datasources."datasources\.yaml".datasources[0]    name=Prometheus \
            --set datasources."datasources\.yaml".datasources[0]    type=prometheus \
            --set datasources."datasources\.yaml".datasources[0]    url=http://prometheus-server.prometheus.svc.cluster.local \
            --set datasources."datasources\.yaml".datasources[0]    access=proxy \
            --set datasources."datasources\.yaml".datasources[0]    isDefault=true \
            --set service.type=LoadBalancer

Get your 'admin' user password 

    kubectl get secret --namespace grafana grafana \
        -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

The Grafana server can be accessed via port 80 on the following DNS name from within your cluster: grafana.grafana.svc.cluster.local

        

Get the Grafana URL to visit by running these commands in the same shell:

     export SERVICE_IP=$(kubectl get svc --namespace grafana grafana -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
     http://$SERVICE_IP:80

#### Import dashboard

* Cluster Monitoring Dashboard
  - Click ’+’ button on left panel and select ‘Import’.
  - Enter 3119 dashboard id under Grafana.com Dashboard.
  - Click ‘Load’.
  - Select ‘Prometheus’ as the endpoint under prometheus data sources drop down.
  - Click ‘Import’.


* Pods Monitoring Dashboard
  - Click ’+’ button on left panel and select ‘Import’.
  - Enter 6417 dashboard id under Grafana.com Dashboard.
  - Click ‘Load’.
  - Enter Kubernetes Pods Monitoring as the Dashboard name.
  - Click change to set the Unique identifier (uid).
  - Select ‘Prometheus’ as the endpoint under prometheus data sources drop down.s
  - Click ‘Import’.



### Kubernetes Dashboard

* Deploy the Kubernetes Metrics Server

* Deploy the Dashboard

        kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta8/aio/deploy/recommended.yaml


* Create an eks-admin Service Account and Cluster Role Binding

```
cat<<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: eks-admin
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: eks-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: eks-admin
  namespace: kube-system
EOF
```

* Get login token 

        kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep eks-admin | awk '{print $1}')

* Start the kubectl proxy

        kubectl proxy

* Access dashboard via browser
  
        http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#!/login.


* Use token from above to login

* Expose the dashboard to public

```
cat<<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"creationTimestamp":"2020-04-12T12:34:10Z","labels":{"k8s-app":"kubernetes-dashboard"},"name":"kubernetes-dashboard","namespace":"kubernetes-dashboard","resourceVersion":"380715","selfLink":"/api/v1/namespaces/kubernetes-dashboard/services/kubernetes-dashboard","uid":"31489c5f-2dff-4b88-9a36-46b248bf9ce2"},"spec":{"externalTrafficPolicy":"Cluster","ports":[{"port":80,"protocol":"TCP","targetPort":8443}],"selector":{"k8s-app":"kubernetes-dashboard"},"sessionAffinity":"None","type":"LoadBalancer"},"status":{"loadBalancer":{}}}
  creationTimestamp: "2020-04-12T13:23:52Z"
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
  resourceVersion: "402339"
  selfLink: /api/v1/namespaces/kubernetes-dashboard/services/kubernetes-dashboard
  uid: d29bab71-d159-4c33-9ba1-00f05138ecb6
spec:
  externalTrafficPolicy: Cluster
  ports:
  - nodePort: 30556
    port: 443
    protocol: TCP
    targetPort: 8443
  selector:
    k8s-app: kubernetes-dashboard
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:{}
```




<file_sep>+++
title = "Windows Terminal"
description = "Empower Windows with Terminal"
+++


### Windows Terminal

> The Windows Terminal is a modern, fast, efficient, powerful, and productive terminal application for users of command-line tools and shells like Command Prompt, PowerShell, and WSL. Its main features include multiple tabs, panes, Unicode and UTF-8 character support, a GPU accelerated text rendering engine, and custom themes, styles, and configurations.

#### Prerequisite
- PowerShell 7
  - Install latest PowerShell (version 7.x) from Windows Store
- Scoop 
  - [Scoop](https://scoop.sh) can be found here or simply execute the command below
    
    ```powershell
    Invoke-Expression (New-Object System.Net.WebClient).DownloadString('https://get.scoop.sh')

    # or shorter
    iwr -useb get.scoop.sh | iex
    ```

#### Install Windows Terminal

This is an open source project and we welcome community participation. To participate please visit https://github.com/microsoft/terminal

#### Install Nerd font

- [Nerd Font](https://www.nerdfonts.com/font-downloads)
- Download Meslo Nerd Font
- Install font file



#### Update Setting

- Settings > Startups
    - Profile -> PowerShell ( Default is Windows PowerShell )
    - Launch mode -> Maximised

- Update Setting file

    - Hide the proflle Windows PowerShell & move the PowerShell to the top

    ```json
    "list": [
        {
                "guid": "{574e775e-4f2a-5b96-ac1e-a2962a402336}",
                "hidden": false,
                "name": "PowerShell",
                "source": "Windows.Terminal.PowershellCore"
        },
        ...
        {
                "commandline": "powershell.exe",
                "font": 
                {
                    "face": "MesloLGM NF"
                },
                "guid": "{61c54bbd-c2c6-5271-96e7-009a87ff44bf}",
                "hidden": true,
                "name": "Windows PowerShell"
        } 
    ```

    - Create a customzed color scheme

    ```json
    {
        "background": "#001D26",
        "black": "#282C34",
        "blue": "#61AFEF",
        "brightBlack": "#9aabc5",
        "brightBlue": "#61AFEF",
        "brightCyan": "#56B6C2",
        "brightGreen": "#98C379",
        "brightPurple": "#C678DD",
        "brightRed": "#E06C75",
        "brightWhite": "#DCDFE4",
        "brightYellow": "#E5C07B",
        "cursorColor": "#FFFFFF",
        "cyan": "#56B6C2",
        "foreground": "#DCDFE4",
        "green": "#98C379",
        "name": "One Half Dark (mod)",
        "purple": "#C678DD",
        "red": "#E06C75",
        "selectionBackground": "#FFFFFF",
        "white": "#DCDFE4",
        "yellow": "#E5C07B"
    }
    ```

- Settings > Default > Appearence
  - Color scheme : One Half Dark (mod)
  - Font face:  MesloLGM NF
  - Acrylic opacity: 60% 


#### Git

```
winget install -e --id Git.Git
```

#### Other tools

```
scoop install curl sudo jq yq
scoop install neovim gcc 
scoop install 7zip bat 
```

#### Setup User Profile

- Create user profile folder

```cmd
mkdir ~\.config\powershell
```

- Create a launch script

```powershell
type nul > $env:USERPROFILE\.config\powershell\profile.ps1
```

- Add alias to `$env:USERPROFILE\.config\powershell\profile.ps1`

```powershell
# Alias
Set-Alias vi nvim
Set-Alias ll ls
Set-Alias g git
Set-Alias grep findstr
Set-Alias tig $env:USERPROFILE\app\git\usr\bin\tig.exe
Set-Alias less $env:USERPROFILE\app\git\usr\bin\less.exe
```

- Update built-in profile

```powershell
# Get all profiles
$PROFILE | Get-Member -Type NoteProperty

# If the file of $PROFILE.CurrentUserCurrentHost does not exist
type nul > $PROFILE.CurrentUserCurrentHost
```

- Copy content below to `$PROFILE.CurrentUserCurrentHost`

```
. $env:USERPROFILE\.config\powershell\profile.ps1
```


#### Oh My Posh

- Installation

  [Please follow latest instruction](https://ohmyposh.dev/docs/migrating)


- Create customized theme file `sample.omp.json`

```
{
  "$schema": "https://raw.githubusercontent.com/JanDeDobbeleer/oh-my-posh/main/themes/schema.json",
  "final_space": false,
  "osc99": true,
  "blocks": [
    {
      "type": "prompt",
      "alignment": "left",
      "segments": [
        {
          "type": "shell",
          "style": "diamond",
          "leading_diamond": "╭─",
          "trailing_diamond": "",
          "foreground": "#ffffff",
          "background": "#0077c2",
          "properties": {
          }
        },
        {
          "type": "root",
          "style": "diamond",
          "leading_diamond": "",
          "trailing_diamond": "",
          "foreground": "#FFFB38",
          "background": "#ef5350",
          "properties": {
            "root_icon": "\uf292",
            "prefix": "<parentBackground>\uE0B0</> "
          }
        },
        {
          "type": "path",
          "style": "powerline",
          "powerline_symbol": "\uE0B0",
          "foreground": "#E4E4E4",
          "background": "#444444",
          "properties": {
            "style": "full",
            "enable_hyperlink": true
          }
        },
        {
          "type": "git",
          "style": "powerline",
          "powerline_symbol": "\uE0B0",
          "foreground": "#011627",
          "background": "#FFFB38",
          "background_templates": [
            "{{ if or (.Working.Changed) (.Staging.Changed) }}#ffeb95{{ end }}",
            "{{ if and (gt .Ahead 0) (gt .Behind 0) }}#c5e478{{ end }}",
            "{{ if gt .Ahead 0 }}#C792EA{{ end }}",
            "{{ if gt .Behind 0 }}#C792EA{{ end }}"
          ],
          "properties": {
            "branch_icon": "\ue725 ",
            "fetch_status": true,
            "fetch_upstream_icon": true,
            "template": "{{ .HEAD }} {{ if .Working.Changed }}{{ .Working.String }}{{ end }}{{ if and (.Working.Changed) (.Staging.Changed) }} |{{ end }}{{ if .Staging.Changed }}<#ef5350> \uF046 {{ .Staging.String }}</>{{ end }}"
          }
        }
      ]
    },
    {
      "type": "prompt",
      "alignment": "right",
      "segments": [
        {
          "type": "node",
          "style": "diamond",
          "leading_diamond": " \uE0B6",
          "trailing_diamond": "\uE0B4",
          "foreground": "#3C873A",
          "background": "#303030",
          "properties": {
            "prefix": "\uE718 ",
            "postfix": "",
            "display_package_manager": true,
            "yarn_icon": " <#348cba></>",
            "npm_icon": " <#cc3a3a></> "
          }
        },
        {
          "type": "time",
          "style": "diamond",
          "invert_powerline": true,
          "leading_diamond": " \uE0B6",
          "trailing_diamond": "\uE0B4",
          "background": "#40c4ff",
          "foreground": "#ffffff",
          "properties": {
            "prefix": " \uf5ef ",
            "postfix": " "
        }
        }
      ]
    },
    {
      "type": "prompt",
      "alignment": "left",
      "newline": true,
      "segments": [
        {
          "type": "text",
          "style": "plain",
          "foreground": "#21c7c7",
          "properties": {
            "prefix": "",
            "postfix": "",
            "text": "╰─"
          }
        },
        {
          "type": "exit",
          "style": "plain",
          "foreground": "#e0f8ff",
          "properties": {
            "prefix": "\u276F",
            "display_exit_code": false,
            "always_enabled": true,
            "error_color": "#ef5350"
          }
        }
      ]
    }
  ]
}

```


- Update user profile

  - Add content to `$env:USERPROFILE\.config\powershell\profile.ps1`

  ```powershell

  # Prompt
  Import-Module posh-git
  $omp_config = "$env:USERPROFILE\sample.omp.json"
  oh-my-posh --init --shell pwsh --config $omp_config | Invoke-Expression
  ```


#### Posh-Git 

```
Install-Module posh-git -Repository PSGallery -Force
```

#### Terminal Icons

- Installation

```powershell
 Install-Module -Name Terminal-Icons -Repository PSGallery -Force
```

- Update user profile
  - Add content to `$env:USERPROFILE\.config\powershell\profile.ps1`

  ```powershell
  Import-Module -Name Terminal-Icons
  ```

#### PSReadLine

- Installation


```powershell
Install-Module -Name PSReadLine  -AllowPrerelease -Scope CurrentUser -Force -SkipPublisherCheck
```

- Update user profile
  - Add content to `$env:USERPROFILE\.config\powershell\profile.ps1`

  ```powershell
  # PSReadLine
  Set-PSReadLineOption -EditMode Emacs
  Set-PSReadLineOption -BellStyle None
  Set-PSReadLineKeyHandler -Chord 'Ctrl+d' -Function DeleteChar
  Set-PSReadLineOption -PredictionSource History
  ```

#### FZF

- Installation

```powershell
scoop install fzf
Install-Module -Name PSFzf -Scope CurrentUser -Force
```

- Update user profile
  - Add content to `$env:USERPROFILE\.config\powershell\profile.ps1`
  ```powershell
  # Fzf
  Import-Module PSFzf
  Set-PsFzfOption -PSReadlineChordProvider 'Ctrl+f' -PSReadlineChordReverseHistory 'Ctrl+r'
  ```
#### Z

```
Install-Module -Name z --Respository SGallery -Force
```

#### Exmple of launch script

```powershell
# Alias
Set-Alias vi nvim
Set-Alias ll ls
Set-Alias g git
Set-Alias grep findstr
Set-Alias tig $env:USERPROFILE\app\git\usr\bin\tig.exe
Set-Alias less $env:USERPROFILE\app\git\usr\bin\less.exe

# Prompt
Import-Module posh-git
$omp_config = "$env:USERPROFILE\sample.omp.json"
oh-my-posh --init --shell pwsh --config $omp_config | Invoke-Expression

# Terminal Icons
Import-Module -Name Terminal-Icons

# PSReadLine
Set-PSReadLineOption -EditMode Emacs
Set-PSReadLineOption -BellStyle None
Set-PSReadLineKeyHandler -Chord 'Ctrl+d' -Function DeleteChar
Set-PSReadLineOption -PredictionSource History

# Fzf
Import-Module PSFzf
Set-PsFzfOption -PSReadlineChordProvider 'Ctrl+f' -PSReadlineChordReverseHistory 'Ctrl+r'

# Customized functions 
# Add your functions here

```<file_sep>+++

title = "Scheduled task with window service"
description="How to build a configurable scheduled task on Window Service"
+++

## Problem

> *Set up the some scheduled tasks running in the backgroud to take care of data update or sync for every 15 mins, or everyday or every week*

## Solution

### Option 1

## Windows Task Scheduler

* Click the Start button.
* Click Control Panel.
* Click System and Maintenance.
* Click Administrative Tools.
* Double-click Task Scheduler.

### Option 2

## Use Window Service as task scheduler

### Overview of design

**The design here is a simplified version, which I built for previous projects. In the real world, I need to tailor it for different project with different purpose, but fundamental design of architect is the same. IMO, the design below can support most cases, which needs scheduled backgroud task service.**

```ini

        +----------------------------------+          Register as
        |     System.ServiceProces  -----  |--------o Window Service
        +----------------------------------+
        |  +----------------------------+  |
        |  |   Thread (Infinite)        |  | 
        |  +----------------------------+  |
        |  |  +----------------------+  |  |
        |  |  | MySchedulerService   |  |  | 
        |  |  +----------------------+  |  |
        |  |  | while ( true )       |  |  |
        |  |  | {                    |  |  |
        |  |  |    Task.Process()    |  |  |
        |  |  | }                    |  |  | 
        |  |  +----------------------+  |  | 
        |  +----------------------------+  |
        +----------------------------------+ 

            +------------+
            | ITask      |
            +------------+
            | Process()  |
            +------------+
                 /|\
                  |
        +------------------+
        |      BaseTask    |  -----------o Customized Task inherit BaseTask   
        +------------------+
        | lastProcessTime  | ------------o Last process time 
        | intervalTime     | ------------o Customize for next process time 
        | IsReadyProcess() | ------------o Check taks is ready to process 
        +------------------+

```

### Create customized Window Service

### Use ServiceProcess to create Window Service

* The System.ServiceProcess allow you to implement, install, and control Windows service applications. Services are long-running executables that run without a user interface. 
*  The project must have main method the entry point

```cs
namespace MyScheduler
{
    using System.ServiceProcess;

    class Program
    {
        static void Main(string[] args)
        {
            #if DEBUG
                MySchedulerService debugService = new MySchedulerService();
                debugService.onDebug();
                System.Threading.Thread.Sleep(System.Threading.Timeout.Infinite);
            #else
                ServiceBase.Run(new ServiceBase[] { new MySchedulerService() });
            #endif
        }
    }
}

```

### Create customized ServiceBase

* Implementing a service involves inheriting from the ServiceBase class and defining specific behavior to process when start, stop, pause, and continue commands are passed in, as well as custom behavior and actions to take when the system shuts down.

```cs

namespace MyScheduler
{
    using System;
    using System.IO;
    using System.Collections.Generic;
    using System.Threading;

    public partial class MySchedulerService : System.ServiceProcess.ServiceBase
    {
        public const string START_SERVICE = "start.service";
        public const string STOP_SERVICE = "stop.service";

        public void onDebug()
        {
            OnStart(null);
        }

        protected override void OnStart(string[] args)
        {
            System.IO.File.Create(AppDomain.CurrentDomain.BaseDirectory 
                    + TART_SERVICE);
            ThreadStart tsTask = new ThreadStart(TaskLoop);
            Thread rtkTask = new Thread(tsTask);
            rtkTask.Start();
        }

        static void TaskLoop()
        {
            while (true)
            {
                // Exceute scheduled task
                ScheduledTask();
                // Then, wait for certain time interval
                System.Threading.Thread.Sleep(TimeSpan.FromMilliseconds(500));
            }
        }

        static void ScheduledTask()
        {
            // Task code which is executed periodically
            try
            {
                // Call customized tasks
                var types = Assembly.GetExecutingAssembly()
                                        .GetExportedTypes()
                                        .Where(p => typeof(ITask)
                                             .IsAssignableFrom(p.BaseType));

                foreach (var t in types)
                {
                    var task = (ITask)Activator.CreateInstance(t);
                    if (taskSettings.Keys.Contains(t.Name))
                    {
                        task.TaskSetting = taskSettings[t.Name];
                    }
                    tasks.Add(task);
                }
            }
            catch (Exception e)
            {
                // TO Something here               
            }
        }

        protected override void OnStop()
        {
            // Insert code here to close all the open IO or conection.
            System.IO.File.Create(AppDomain.CurrentDomain.BaseDirectory
                         + STOP_SERVICE);
        }

        private void InitializeComponent()
        {
            this.ServiceName = "MySchedulerService";
        }

        protected override void Dispose(bool disposing)
        {
            OnStop();
            base.Dispose(disposing);
        }
    }
}

```

### Task interface and class 

* interface ITask has only one method 

```cs
namespace MyScheduler
{
    public interface ITask
    {
        void Process();
    }
}
```

* BaseTask 

```cs
namespace MyScheduler
{
    using System;

    public class BaseTask : ITask
    {
        protected DateTime? lastProcessTime = null;

        // interval time uses second as unit
        // e.g. 1 min of intervaling time is 60 
        protected int intervalTime = 0 ;  

        protected bool IsReadyToProcess()
        {
            bool isReadyToProcess = true;

            if (lastProcessTime.HasValue)
            {
                if (lastProcessTime.Value.AddSeconds(intervalTime) > DateTime.Now)
                {
                    isReadyToProcess = false;
                }
            } 
            return isReadyToProcess;
        }

        public virtual void Process()
        {
            throw new NotImplementedException();
        }
    }
}
```

* Sample Task1 and Task2

**Task1**

```cs
namespace MyScheduler
{
    using System;
    using System.Linq;
    using System.Data.Entity;

    public class Task1 : BaseTask
    {
        public override void Process()
        {
            if (base.IsReadyToProcess())
            {                
                System.IO.File.Create(AppDomain.CurrentDomain.BaseDirectory
                         + "Task-1-" + DateTime.Now.ToString("dd-MM-yyyy"));
            }
        }
    }
}
```
**Task2**

```cs
namespace MyScheduler
{
    using System;
    using System.Linq;
    using System.Data.Entity;

    public class Task2 : BaseTask
    {
        public override void Process()
        {
            if (base.IsReadyToProcess())
            {                
                System.IO.File.Create(AppDomain.CurrentDomain.BaseDirectory 
                        + "Task-2-" 
                        + DateTime.Now.ToString("dd-MM-yyyy"));
            }
        }
    }
}
```

## Create Window Service installer

* In Solution Explorer, double-click MyScheduledService.cs.
* In the Code Editor window, right-click `Design View`, and then click `Properties`
* In the `Properties` pane, click the `Add Installer` link.
* In the Properties pane for MyScheduledServiceInstaller, change the `ServiceNameproperty` to `MyScheduledService`.
* In the Code Editor window in `Design view`, click `MyScheduledServiceProcessInstaller`.
* In the Properties pane, change the `Account` property to `LocalSystem` (The `LocalService` and `NetworkService` values are available only in Microsoft Windows XP).

**PreInstaller**

* It inherits from System.Configuration.Install.Installer. This is the base class for all custom installers in the .NET Framework. Installers are components that help install applications on a computer.

```cs
namespace MyScheduler
{
    [RunInstaller(true)]
    public partial class ProjectInstaller : System.Configuration.Install.Installer
    {
        public ProjectInstaller()
        {
            InitializeComponent();
        }

        private void MySchedulerServiceInstaller_AfterInstall(
                object sender, InstallEventArgs e)
        {
        }
    }
}
```

## Install Window Service 

```bash
C:\Windows\Microsoft.Net\Framework\v4.0.30319\InstallUtil.exe /i <app_path>\MyScheduler.exe 
```

## Uninstall Window Service 

```bash
C:\Windows\Microsoft.Net\Framework\v4.0.30319\InstallUtil.exe /u <app_path>\MyScheduler.exe 
```<file_sep>+++
title = "Cloud"
weight = 3  
+++

{{%children style="card" description="true" %}}

<file_sep>#!/bin/bash

VPC_ID="vpc-04c97179894199703"
AWS_REGION="ap-southeast-2"

# Create a NACL for private subnet
NACL_PRVSUB_ID=$(aws ec2 create-network-acl \
  --vpc-id $VPC_ID \
  --query 'NetworkAcl.{NetworkAclId:NetworkAclId}' \
  --output text \
  --region $AWS_REGION)

echo $NACL_PRVSUB_ID

# Create NACL rule
aws ec2 create-network-acl-entry \
  --network-acl-id $NACL_PRVSUB_ID \
  --ingress --rule-number 100 \
  --protocol udp --port-range \
  From=53,To=53 --cidr-block 0.0.0.0/0 \
  --rule-action allow

# Describe NACL
aws ec2 describe-network-acl \
  --network-acl-id $NACL_PRVSUB_ID \
  --output text

<file_sep>+++
title = "AWS: S3 - 2"
description = "S3 Part 2 - Access & Management"
weight=7
+++

## S3 Part 2

### Access 

#### Access status

The list buckets view shows whether your bucket is publicly accessible. Amazon S3 labels the permissions for a bucket as follows:

* Public – Everyone has access to one or more of the following: List objects, Write objects, Read and write permissions.

* Objects can be public – The bucket is not public, but anyone with the appropriate permissions can grant public access to objects.

* Buckets and objects not public – The bucket and objects do not have any public access.

* Only authorized users of this account – Access is isolated to IAM users and roles in this account and AWS service principals because there is a policy that grants public access.


### Management 

Bucket and object permissions are independent of each other. An object does not inherit the permissions from its bucket. For example, if you create a bucket and grant write access to a user, you can't access that user’s objects unless the user explicitly grants you access.

Each permission you grant for a user or a group adds an entry in the ACL that is associated with the object. The ACL lists grants, which identify the grantee and the permission granted. ACLs are resource-based access policies that grant access permissions to buckets and objects.


### Block public access settings

S3 Block Public Access provides four settings. You can apply these settings in any combination to individual access points, buckets, or entire AWS accounts. If you apply a setting to an account, it applies to all buckets and access points that are owned by that account. Similarly, if you apply a setting to a bucket, it applies to all access points associated with that bucket.

#### ACLs

Amazon S3 considers a bucket or object ACL public if it grants any permissions to members of the predefined AllUsers or AuthenticatedUsers groups.

#### Policies

When evaluating a bucket policy, Amazon S3 begins by assuming that the policy is public.  It then evaluates the policy to determine whether it qualifies as non-public. To be considered non-public, a bucket policy must grant access only to fixed values.

#### Access points

Amazon S3 evaluates block public access settings slightly differently for access points compared to buckets. The rules that Amazon S3 applies to determine when an access point policy is public are generally the same for access points as for buckets, except in the following situations:

* An access point that has a VPC network origin is always considered non-public, regardless of the contents of its access point policy.

* An access point policy that grants access to a set of access points using s3:DataAccessPointArn is considered public. 

#### Permissions

| Operation	| Required permissions
|:----|--- 
| GET bucket policy status	|  s3:GetBucketPolicyStatus
| GET bucket Block Public Access settings	|  s3:GetBucketPublicAccessBlock
| PUT bucket Block Public Access settings	|  s3:PutBucketPublicAccessBlock
| DELETE bucket Block Public Access settings	|  s3:PutBucketPublicAccessBlock
| GET account Block Public Access settings	|  s3:GetAccountPublicAccessBlock
| PUT account Block Public Access settings	|  s3:PutAccountPublicAccessBlock
| DELETE account Block Public Access settings	|  s3:PutAccountPublicAccessBlock
| PUT access point Block Public Access settings	|  s3:PutAccessPointPublicAccessBlock





<file_sep>+++
date = "2011-03-09T10:59:31+11:00"
title = "CMMI 2.0"
description="CMMI Version 2.0"
weight=1
+++

## CMMI 2.0

### Guilding Principles

* Focus of delivering business value
* Focus on business performance and results
* Flexible, scalable architecture
* Reduce redundancy
* Use of plain language
* Increased use of graphics and iconography for understanding and adoption

### Key Improvements

* Demonstrate the value and ROI of aclopting CMMI
  
    Performance capabilities built-in at every level of the model will help organizations to:
    * Understand performance needs	
    * Track, measure and achieve
    * Establish performance goals	those goals

* Improve the overall value for CMMI appraisals and lower time, effort and cost of the appraisal process
  
    * New appraisal method intended to improve confidence and reliability of results lower total life cycle costs of appraisals by decreasing the appraisal preparation fo the appraised organization.
    * Organizations can extend the validity of benchmark appraisals through the lighter-weight Sustainment appraisal.

* Keep CMMI current and up-to-date with latest trend methodologies used in the market

    * Scalable architecture platform to include additional method guidance, such as built-in agile with Scrum guidance.
    * Ability to add new content additions, such as Safety and Security, address critical business needs

    
* Make CMMI easier to use and more user friendly

    * Non-technical language makes it easier for users to read and understand the model.
    * Online platform allows users to tailor the model to fit specific organizational needs.
    * Tools provide guidance for the successful adoption of CMMI & transition to
    * Model, training, and usage guidance will be translated into several languages.

### Product Suite

#### Model
  
Clear pathway to performance improvement. Simplified for accelerated adoption.

#### Appraisal Method

A new appraisal method helps to increase reliability while reducing overall cost.

#### Training & Certification

Updated training has modular components with virtual and in-person options. The training is more learning objective oriented.

#### Systems & Tools

Redesigned system to access online models and resources.

#### Adoption Guidance

Guidance to help new adopters and users get started with CMMI V2.0. Provides a smooth transition from CMMI V1.3 to V2.0

### Transformation

CMMI V2.0 transforms to a business performance model

The addition of the intent and value statements

* The Intent statement provides insight into why the practice area is important to take into consideration in your process
* The Value statement explains the business impact achieved when the intent of the practice area and individual practices are met

### Appraisal Changes

* Simplified presentation of information
* Types of appraisals
    * Benchmark
    * Sustainment
    * Evaluation
    * Action Plan Reappraisal
* Model Views
    * Benchmark
    * Custom
* Using Building and Sustaining Capability practice areas (GOV, II)
* Appraisal Team Members
    * ATM Qualification: CMMI Associate
    * High Maturity: Complete High Maturity Concepts
* Random Sampling
* Data adequacy and sufficiency
* Characterization
* Rating Process
* Performance Report
* Validity period for appraisals



### Capability Areas

#### DOING 

Doing includes Capability Areas for producing, buying, and delivering quality solutions.
Ensuring Quality (ENQ) – Helps to improve product and service quality.

Requirements Development and Management enables developing and keeping updated a common understanding of needs and expectations for the solution.

* Intent: Elicit requirements, ensure common understanding by stakeholders, and align requirements, plans, and work products.
* Value: Ensures that customer’s needs and expectations are satisfied

Process Quality Assurance ensures the process is followed and quality solutions are produced

* Intent: Identify causes of selected outcomes and take action to either prevent recurrence of undesirable outcomes or ensure recurrence of positive outcomes.
* Value: Addressing root cause issues eliminates rework and directly improves quality and productivity.

Verification and Validation ensures requirements are met and the solution functions as intended in the target environment.
Intent: Verification and validation includes activities that confirm selected solutions and components meet their requirements and validate selected solutions and components fulfill their intended use in their target environment
Value: Verification and validation of selected solutions and components throughout the project increases the likelihood that the solution will satisfy the customer.

Peer Reviews identify solution defects or issues.

* Intent: Identify and address work product issues through reviews by the producer’s peers or Subject Matter Experts (SMEs).
* Value: Reduce cost and rework by uncovering issues or defects early.

Engineering the Developing Products (EDP) (Development View)

Technical Solution focuses on designing and building products and product components.
Intent: Design and build solutions that meet customer requirements.
Value: Provides a cost-effective design and solution that meets customer requirements and reduces rework.

Product Integration covers the assembly of the products and product components and their delivery to the customer and ensures inclusion of required functionality and quality characteristics.

* Intent: Integrate and deliver the solution that addresses functionality and quality requirements.
* Value: Increases customers’ satisfaction by giving them a solution that meets or exceeds their functionality and quality requirements.

Delivering and Managing Services (Services View)

Service Delivery Management includes delivering services in accordance with the established service level agreements.

* Intent: Deliver services and manage the service delivery system.
* Value: Increase customer satisfaction by delivering services that meet or exceed customer expectations.

Strategic Service Management includes developing and keeping a portfolio of updated standard services that are compatible with strategic needs and plans.

* Intent: Develop and deploy standard services that are compatible with strategic business needs and plans.
* Value: Increases likelihood of meeting business objectives by aligning standard services with customer needs.

Supplier Agreement Management involves: developing and keeping updated the supplier agreement and ensuring that the supplier and the buyer perform according to the terms of the supplier agreement

* Intent: Establish an agreement with selected suppliers, ensure that the supplier and the acquirer perform according to the terms over the course of the agreement, and evaluate the supplier’s deliverables.
* Value: Provides an explicit understanding between the acquirer and supplier to maximize the success of agreed-on efforts to deliver a supplier deliverable.

#### MANAGING 

Managing includes capability areas for planning and managing work and the workforce.
Planning and Managing Work – involves determining the amount of work that needs to be done, planning and scheduling the work, and then ensuring the work is being done in accordance with the plans and schedules. It also ensures that resources are adequate to meet the plan and schedule.

Estimating includes forecasting the size, effort, and cost for the work required to develop, acquire, or deliver the solution

* Intent: Estimate the size, effort, duration, and cost of the work and resources needed to develop, acquire, or deliver the solution.
* Value: Estimation provides a basis for making commitments, planning, and reducing uncertainty, which allows for early corrective actions and increases the likelihood of meeting objectives.

Planning involves: Using the estimates to develop a work plan, schedule, and budget; Determining the necessary resources to accomplish the plan, within schedule and budget; Obtaining commitment to the work plan from stakeholders

* Intent: Develop plans to describe what is needed to accomplish the work within the standards and constraints of the organization, including the: Budget; Schedule; Resource demand, capacity and availability; Quality; Functionality requirements; Risks and opportunities
    Plans also describe: The work to be performed; Applicable organizational set of standard processes, assets, and tailoring guidelines; Dependencies; Who performs the work; Relationships with other plans; Stakeholders and their role
* Value: Optimizes cost, functionality, and quality to increase the likelihood of meeting objectives.

Monitor and Control provides an understanding of progress so appropriate corrective actions can be taken when performance deviates significantly from the plan, schedule, and budget.

* Intent: Provide an understanding of the project progress so appropriate corrective actions can be taken when performance deviates significantly from plans.
* Value: Increases the probability of meeting objectives by taking early actions to adjust for significant performance deviations.

Managing Business Resilience addresses the ability to anticipate, prepare for, and respond to interruptions in order to continue operations. It involves identifying, evaluating, prioritizing and handling risks. It ensures timely and effective resolution and prevention of interruptions to minimize the impact on business operations and ensures the best possible level of service quality. It addresses defining a minimum set of critical functions that must continue in the event of significant interruption of normal operations.

Risk and Opportunity Management includes: Identifying threats and opportunities; Evaluating their likelihood of occurrence and impact; Mitigating potential threats; Leveraging potential opportunities
* Intent: Identify, record, analyze, and manage potential risks or opportunities.
* Value: Mitigate adverse impacts or capitalize on positive impacts to increase the likelihood of meeting objectives.

Incident Resolution and Prevention includes: Identifying actual and potential incidents that may impact delivery; Establishing the approach for addressing incidents as they occur; Analyzing incidents to prevent recurrence

* Intent: Resolve and prevent disruptions promptly to sustain service delivery levels.
* Value: Minimize the impact of disruptions to meet objectives and customer commitments more effectively.

Managing the Workforce addresses the way an organization develops and retains the human resources needed to perform current and future work.

Organizational Training provides a strategy and capability for training to support the organization’s strategic business objectives, meet common tactical needs, and deliver training across the organization.

* Intent: Develop the skills and knowledge of personnel so they perform their roles efficiently and effectively.
* Value: Enhances individuals’ skills and knowledge to improve organizational work performance.
    Practice Summary

#### ENABLING 

Enabling focuses on analyzing causes, making decisions, maintaining integrity of work products, and communicating to stakeholders.
Supporting Implementation involves identifying and addressing the causes of selected outcomes, creating a decision-making approach and structure, maintaining the integrity of work products, and fostering communication and coordination among stakeholders.

Causal Analysis and Resolution identifies causes of selected outcomes and acts to either prevent reoccurrence of undesirable outcomes or ensure reoccurrence of positive outcomes.

* Intent: Identify causes of selected outcomes and take action to either prevent recurrence of undesirable outcomes or ensure recurrence of positive outcomes
* Value: Addressing root cause issues eliminates rework and directly improves quality and productivity.

Decision Analysis and Resolution aids in making decisions using criteria-based evaluation of alternatives and recording the results.

* Intent: Make and record decisions using a recorded process that analyzes alternatives
* Value:  Increases the objectivity of decision making and the probability of selecting the optimal solution.

Configuration Management establishes and maintains the integrity of work products using configuration identification, control, and audits.

* Intent: Manage the integrity of work products using configuration identification, version control, change control, and audits.
* Value: Reduces loss of work and increases the ability to deliver the correct version of the solution to the customer.

#### IMPROVING 

Improving involves developing, managing, and improving processes and their related assets with a primary focus on improving organizational performance.
Sustaining Habit and Persistence ensures that processes are persistent and habitually performed and sustained throughout the organization and effectively contribute to meeting business performance objectives.

Governance provides guidance to senior management on their role in ensuring that work is performed in a way that is relevant and important to the business and organization.

* Intent: Provides guidance to senior management on their role in the sponsorship and governance of process activities.
* Value: Minimizes the cost of process implementation, increases the likelihood of meeting objectives, and ensures that the implemented processes support and contribute to the success of the business.

Implementation Infrastructure provides a framework that ensures the processes of an organization are persistently used and improved.

* Intent: Ensure that the processes important to an organization are persistently and habitually used and improved.
* Value: Sustains the ability to consistently achieve goals and objectives efficiently and effectively.
    

Improving Performance focuses on measuring, analyzing and understanding an organization’s or project’s capability and performance along with their process improvement priorities and infrastructure needs. Once this is understood, the organization or project can identify performance and process improvement actions and assets needed to continually improve capability and performance.

Process Management develops capabilities and improves performance though planning, implementing, and deploying improvements based on a thorough understanding of the current strengths and weaknesses of the organization’s processes and process assets.

* Intent: Manages and implements the continuous improvement of processes and infrastructure to: Support accomplishing business objectives; Identify and implement the most beneficial process improvements; Make the results of process improvement visible, accessible, and sustainable
* Value: Ensures that processes, infrastructure, and their improvement contribute to successfully meeting business objectives.

Process Asset Development develops and keeps updated a usable set of organizational processes and process assets for performing the work.

* Intent: Develop and keep updated the process assets necessary to perform the work.
* Value: Provides a capability to understand and repeat successful performance

Managing Performance and Measurement involves:
Ensuring that benefits and business objectives are the leading factors driving performance and improvement
Changing the paradigm: From process improvement leads to performance improvement; Performance needs are the primary drivers of process improvements; Using the results of measurement and analysis to manage and control performance at various work and business levels

* Intent: Manage performance using measurement and analysis to achieve business objectives.
* Value: Maximizes business return on investment by focusing management and improvement efforts on cost, schedule, and quality performance.

Selecting and Managing Suppliers (Supplier Management View) establishes the buyer and supplier partnership to ensure that quality solutions are delivered to the customer and end user

Supplier Source Selection involves: Selecting one or more suppliers to deliver the solution; Preparing a solicitation package; Evaluating the supplier’s solution and managing selected connections of that solution

* Intent: Develop and keep updated a package of materials used to seek proposals from potential suppliers and select one or more suppliers to deliver the solution.
* Value: Improves the ability to select the most qualified suppliers to deliver solutions.
    




<file_sep>+++
title = "C# - .Net Core 6"
description = ".Net Core 6 - Part 1"
draft=true
+++


## .Net Core 6
<file_sep>+++
title = "AWS : CLI - 1"
description = "AWS CLI & Sample"
weight=2
+++


### AWS CLI

Amazon Virtual Private Cloud (Amazon VPC) enables you to launch AWS resources into a virtual network that you've defined. This virtual network closely resembles a traditional network that you'd operate in your own data center, with the benefits of using the scalable infrastructure of AWS.

### CLI installation (Version 1)

#### Install Python 3

* Python 2.7 is no longer supported. Please install python3

        sudo yum install python3
        pip3 --version

* 




### Profile Setup

A named profile is a collection of settings and credentials that you can apply to a AWS CLI command. When you specify a profile to run a command, the settings and credentials are used to run that command. 

* `~/.aws/credentials`

        [default]
        aws_access_key_id=your_aws_access_key_id
        aws_secret_access_key=your_aws_secret_access_key

* `~/.aws/config`

        [default]
        region=us-west-2
        output=json

* The AWS CLI supports using any of multiple named profiles that are stored in the config and credentials files. 

* `~/.aws/credentials` (Linux / Mac)

        [default]
        aws_access_key_id=your_aws_access_key_id
        aws_secret_access_key=your_aws_secret_access_key

        [user1]
        aws_access_key_id=your_aws_access_key_id
        aws_secret_access_key=your_aws_secret_access_key

* `~/.aws/config` (Linux & Mac)

        [default]
        region=us-west-2
        output=json

        [profile user1]
        region=us-east-1
        output=text


### Samples

#### EC2

* Get EC2 instances 

        aws ec2 describe-instances \
        --filters "Name=instance-type,Values=t2.micro" \
        --query "Reservations[].Instances[].InstanceId"  \
        --profile user1

* Stop EC2 instance

        aws ec2 stop-instances --instance-ids i-1234567890abcdef0

* Find out all stopped micro EC2 instancs 

        instances=($(aws ec2 describe-instances \
        --filters Name=instance-type,Values=t2.micro \
          Name=instance-state-name,Values=stopped \
        --query "Reservations[].Instances[].InstanceId"  \
        --profile ad1 --output text ))

        for i in $instances; do echo "$i" ; done;


* Find out all unused security group


        comm -23  \
        <(aws ec2 describe-security-groups \
        --profile user1 --query 'SecurityGroups[*].GroupId' \
        --output text \
        | tr '\t' '\n'| sort) \
        <(aws ec2 describe-instances \
        --profile user1   --output text \
        --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' \
        | tr '\t' '\n'  | sort | uniq)

* Find security group by group name 

        aws ec2  describe-security-groups  \
          --filters Name=group-name,Values=AD1_Web_Pub_SG  \
          --query="SecurityGroups[*].{ID:GroupId,Tags:Tags[*]}" \
          --profile user1 

* Search security group by tags

        aws ec2  describe-security-groups \
          --query="SecurityGroups[*].{GroupName:GroupName,ID:GroupId,Tags:Tags[*]}" \
          --profile user1 | grep "your_group_name"

* Search EC2 by specific tag value

        aws ec2 describe-instances \
          --filters "Name=tag-value,Values=AD1_DEV_Web_Pub_Host_A" \
          --profile user1 



#### S3


* List Buckets

        aws s3 ls --profile user1 

* List content within the bucket

        aws s3 ls --profile user1 s3://your_bucket_name --recursive

* Sync everything to curerent directory

        aws s3 sync --profile  user1 s3://your_bucket_name .

* Copy local file to S3 bucket

        aws s3  cp --profile user1 file_name s3://your_bucket_name/file_name









<file_sep>+++
title = "F# Collections 3"
description = "F# Seq, Slice & Map"
weight = 7
+++


## Seq

A sequence is a logical series of elements all of one type. Sequences are particularly useful when you have a large, ordered collection of data but do not necessarily expect to use all of the elements. Individual sequence elements are computed only as required, so a sequence can provide better performance than a list in situations in which not all the elements are used.


```fsharp
// Sequence that has an increment.
seq { 0 .. 10 .. 100 }

// uses the -> operator, which allows you to specify an expression 
// whose value will become a part of the sequence. 
seq { for i in 1 .. 10 -> i * i }


// specify the do keyword, with an optional yield that follows:
seq { for i in 1 .. 10 do yield i * i }

// The 'yield' is implicit and doesn't need to be specified in most cases.
seq { for i in 1 .. 10 do i * i }

// The following code generates a list of coordinate pairs along with 
// an index into an array that represents the grid. 
// Note that the first for expression requires a do to be specified.
let (height, width) = (10, 10)
let sequence = seq {
    for row in 0 .. width - 1 do
        for col in 0 .. height - 1 ->
            (row, col, row*width + col)
    }

for e in sequence do
    printfn "%A" e

// (0, 0, 0)
// (0, 1, 1)
// (0, 2, 2)
// ...
// (0, 9, 9)
// (1, 0, 10)
// (1, 1, 11)
// (1, 2, 12)
// ...
// (9, 9, 99)
```


### yield! keyword


```fsharp
// Repeats '1 2 3 4 5' ten times
seq {
    for _ in 1..10 do
        yield! seq { 1; 2; 3; 4; 5}
}

// When yield! is used in an expression, all other single values 
// must use the yield keyword:
// Combine repeated values with their values
seq {
    for x in 1..10 do
        yield x
        yield! seq { for i in 1..x -> i}
}


// Prime number
// Recursive isprime function.
let isprime n =
    let rec check i =
        i > n/2 || (n % i <> 0 && check (i + 1))
    check 2

let aSequence =
    seq {
        for n in 1..100 do
            if isprime n then
                n
    }

for x in aSequence do
    printfn "%d" x
// output
// 1 2 3 5 7 ...47 53 ... 89 97


// Following example creates a multiplication table that consists of 
// tuples of three elements, each consisting of two factors and the product
let multiplicationTable =
    seq {
        for i in 1..9 do
            for j in 1..9 ->
                (i, j, i*j)
    }



// Yield the values of a binary tree in a sequence.
type Tree<'a> =
   | Tree of 'a * Tree<'a> * Tree<'a>
   | Leaf of 'a

// inorder : Tree<'a> -> seq<'a>
let rec inorder tree =
    seq {
      match tree with
          | Tree(x, left, right) ->
               yield! inorder left
               yield x
               yield! inorder right
          | Leaf x -> yield x
    }

let mytree = Tree(6, Tree(2, Leaf(1), Leaf(3)), Leaf(9))
let seq1 = inorder mytree
printfn "%A" seq1
// outut 
// seq [1; 2; 3; 6; ...]
```

### Functions

```fsharp

// Using Seq.empty, or you can create a sequence of just one 
// specified element by using Seq.singleton.
let seqEmpty = Seq.empty
let seqOne = Seq.singleton 10
printfn "%A" seqOne
// output: seq [10]

// use Seq.init to create a sequence for which the elements are created 
// by using a function that you provide. You also provide a size for 
// the sequence. This function is just like List.init, except that the 
// elements are not created until you iterate through the sequence.
let seqFirst5MultiplesOf10 = Seq.init 5 (fun n -> n * 10)
Seq.iter (fun elem -> printf "%d " elem) seqFirst5MultiplesOf10
// The output is
// 0 10 20 30 40

// Convert an array to a sequence by using a cast.
let seqFromArray1 = [| 1 .. 10 |] :> seq<int>

// Convert an array to a sequence by using Seq.ofArray.
let seqFromArray2 = [| 1 .. 10 |] |> Seq.ofArray

//  using Seq.cast, you can create a sequence from a weakly typed 
// collection, such as those defined in System.Collections. 
open System

let arr = ResizeArray<int>(10)

for i in 1 .. 10 do
    arr.Add(10)

let seqCast = Seq.cast arr


// Seq.unfold generates a sequence from a computation function that takes 
// a state and transforms it to produce each subsequent element in the sequence
let seq1 =
    0 // Initial state
    |> Seq.unfold (fun state ->
        if (state > 20) then
            None
        else
            Some(state, state + 1))

printfn "The sequence seq1 contains numbers from 0 to 20."

for x in seq1 do
    printf "%d " x
// output
// The sequence seq1 contains numbers from 0 to 20.
// 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

let fib =
    (1, 1) // Initial state
    |> Seq.unfold (fun state ->
        if (snd state > 1000) then
            None
        else
            Some(fst state + snd state, (snd state, fst state + snd state)))

printfn "\nThe sequence fib contains Fibonacci numbers."
for x in fib do printf "%d " x
// output
// The sequence fib contains Fibonacci numbers.
// 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597

```


##### Example

```fsharp
// generateInfiniteSequence generates sequences of floating point
// numbers. The sequences generated are computed from the fDenominator
// function, which has the type (int -> float) and computes the
// denominator of each term in the sequence from the index of that
// term. The isAlternating parameter is true if the sequence has
// alternating signs.
let generateInfiniteSequence fDenominator isAlternating =
    if (isAlternating) then
        Seq.initInfinite (fun index ->
            1.0 /(fDenominator index) * (if (index % 2 = 0) then -1.0 else 1.0))
    else
        Seq.initInfinite (fun index -> 1.0 /(fDenominator index))

// The harmonic alternating series is like the harmonic series
// except that it has alternating signs.
let harmonicAlternatingSeries = generateInfiniteSequence (fun index -> float index) true

// This is the series of reciprocals of the odd numbers.
let oddNumberSeries = generateInfiniteSequence (fun index -> float (2 * index - 1)) true

// This is the series of recipocals of the squares.
let squaresSeries = generateInfiniteSequence (fun index -> float (index * index)) false

// This function sums a sequence, up to the specified number of terms.
let sumSeq length sequence =
    (0, 0.0)
    |>
    Seq.unfold (fun state ->
        let subtotal = snd state + Seq.item (fst state + 1) sequence
        if (fst state >= length) then
            None
        else
            Some(subtotal, (fst state + 1, subtotal)))

// This function sums an infinite sequence up to a given value
// for the difference (epsilon) between subsequent terms,
// up to a maximum number of terms, whichever is reached first.
let infiniteSum infiniteSeq epsilon maxIteration =
    infiniteSeq
    |> sumSeq maxIteration
    |> Seq.pairwise
    |> Seq.takeWhile (fun elem -> abs (snd elem - fst elem) > epsilon)
    |> List.ofSeq
    |> List.rev
    |> List.head
    |> snd

// Compute the sums for three sequences that converge, and compare
// the sums to the expected theoretical values.
let result1 = infiniteSum harmonicAlternatingSeries 0.00001 100000
printfn "Result: %f  ln2: %f" result1 (log 2.0)
// output
// Result: 0.693152  ln2: 0.693147

let pi = Math.PI
let result2 = infiniteSum oddNumberSeries 0.00001 10000
printfn "Result: %f pi/4: %f" result2 (pi/4.0)
// output
// Result: 0.785373 pi/4: 0.785398

// Because this is not an alternating series, a much smaller epsilon
// value and more terms are needed to obtain an accurate result.
let result3 = infiniteSum squaresSeries 0.0000001 1000000
printfn "Result: %f pi*pi/6: %f" result3 (pi*pi/6.0)
// output
// Result: 1.644618 pi*pi/6: 1.644934
```


#### Transforming

```fsharp
// Seq.pairwise creates a new sequence in which successive elements of the input
// sequence are grouped into tuples.
let printSeq seq1 = Seq.iter (printf "%A ") seq1; printfn ""
let seqPairwise = Seq.pairwise (seq { for i in 1 .. 10 -> i*i })
printSeq seqPairwise
// output
// (1, 4) (4, 9) (9, 16) (16, 25) (25, 36) (36, 49) (49, 64) (64, 81) (81, 100) 


printfn ""
let seqDelta = Seq.map (fun elem -> snd elem - fst elem) seqPairwise
printSeq seqDelta
// output
// 3 5 7 9 11 13 15 17 19 



// Seq.windowed is like Seq.pairwise, except that instead of producing a 
// sequence of tuples, it produces a sequence of arrays that contain copies 
// of adjacent elements (a window) from the sequence. You specify the number 
// of adjacent elements you want in each array.

let seqNumbers = [ 1.0; 1.5; 2.0; 1.5; 1.0; 1.5 ] :> seq<float>
let seqWindows = Seq.windowed 3 seqNumbers
let seqMovingAverage = Seq.map Array.average seqWindows
printfn "Initial sequence: "
printSeq seqNumbers
// 1.0 1.5 2.0 1.5 1.0 1.5 
printfn "\nWindows of length 3: "
printSeq seqWindows
// [|1.0; 1.5; 2.0|] [|1.5; 2.0; 1.5|] [|2.0; 1.5; 1.0|] [|1.5; 1.0; 1.5|] 

printfn "\nMoving average: "
printSeq seqMovingAverage
// 1.5 1.666666667 1.5 1.333333333 
```

#### Sorting, Comparing & Grouping

```fsharp
// Seq.compareWith function. The function compares successive elements in 
// turn, and stops when it encounters the first unequal pair. Any additional 
// elements do not contribute to the comparison.
let sequence1 = seq { 1 .. 10 }
let sequence2 = seq { 10 .. -1 .. 1 }

// Compare two sequences element by element.
let compareSequences =
    Seq.compareWith (fun elem1 elem2 ->
        if elem1 > elem2 then 1
        elif elem1 < elem2 then -1
        else 0)

let compareResult1 = compareSequences sequence1 sequence2
match compareResult1 with
| 1 -> printfn "Sequence1 is greater than sequence2."
| -1 -> printfn "Sequence1 is less than sequence2."
| 0 -> printfn "Sequence1 is equal to sequence2."
| _ -> failwith("Invalid comparison result.")
// val compareResult1: int = -1


// Seq.countBy takes a function that generates a value called a key for 
// each element. A key is generated for each element by calling this function 
// on each element. Seq.countBy then returns a sequence that contains the key 
// values, and a count of the number of elements that generated each value of 
// the key.

let mySeq1 = seq { 1.. 100 }

let printSeq seq1 = Seq.iter (printf "%A ") seq1

let seqResult =
    mySeq1
    |> Seq.countBy (fun elem ->
        if elem % 3 = 0 then 0
        elif elem % 3 = 1 then 1
        else 2)

printSeq seqResult
// (1, 34) (2, 33) (0, 33)



// Seq.groupBy takes a sequence and a function that generates a key from an 
// element. The function is executed on each element of the sequence. 
// Seq.groupBy returns a sequence of tuples, where the first element of each 
// tuple is the key and the second is a sequence of elements that produce 
// that key.

let sequence = seq { 1 .. 100 }

let printSeq seq1 = Seq.iter (printf "%A ") seq1

let sequences3 =
    sequences
    |> Seq.groupBy (fun index ->
        if (index % 3 = 0) then 0
        elif (index % 3 = 1) then 1
        else 2)

sequences3 |> printSeq
// (1, seq [1; 4; 7; 10; ...]) (2, seq [2; 5; 8; 11; ...]) (0, seq [3; 6; 9; 12; ...])




// Seq.distinct. Or Seq.distinctBy, which takes a key-generating function to 
// be called on each element. The resulting sequence contains elements of the 
// original sequence that have unique keys; later elements that produce a 
// duplicate key to an earlier element are discarded.
let binary n =
    let rec generateBinary n =
        if (n / 2 = 0) then [n]
        else (n % 2) :: generateBinary (n / 2)

    generateBinary n
    |> List.rev
    |> Seq.ofList

printfn "%A" (binary 1024)
// [1; 0; 0; 0; 0; 0; 0; 0; 0; 0; 0]


let resultSequence = Seq.distinct (binary 1024)
printfn "%A" resultSequence
// seq [1; 0]

let inputSequence = { -5 .. 10 }
let printSeq seq1 = Seq.iter (printf "%A ") seq1

printfn "Original sequence: "
printSeq inputSequence
// -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 

printf ""
printfn "\nSequence with distinct absolute values: "
let seqDistinctAbsoluteValue = Seq.distinctBy (fun elem -> abs elem) inputSequence
printSeq seqDistinctAbsoluteValue
// -5 -4 -3 -2 -1 0 6 7 8 9 10 


```


## Slice

In F#, a slice is a subset of any data type. Slices are similar to indexers, but instead of yielding a single value from the underlying data structure, they yield multiple ones. Slices use the .. operator syntax to select the range of specified indices in a data type.






## Map

Immutable maps based on binary trees, where keys are ordered by F# generic comparison. By default comparison is the F# structural comparison function or uses implementations of the IComparable interface on key values.

<file_sep>+++
title = "AWS: VPC - 1"
description = "VPC - Virtual Private Cloud: Subnet, Internet Gateway, Virtual Gateway etc. "
weight=3

+++


## VPC Part 1

Amazon Virtual Private Cloud (Amazon VPC) enables you to launch AWS resources into a virtual network that you've defined. This virtual network closely resembles a traditional network that you'd operate in your own data center, with the benefits of using the scalable infrastructure of AWS.

### Key concepts

* A virtual private cloud (VPC) is a virtual network dedicated to your AWS account.
* A subnet is a range of IP addresses in your VPC.
* A route table contains a set of rules, called routes, that are used to determine where network traffic is directed.
* An internet gateway is a horizontally scaled, redundant, and highly available VPC component that allows communication between instances in your VPC and the internet. It therefore imposes no availability risks or bandwidth constraints on your network traffic.
* A VPC endpoint enables you to privately connect your VPC to supported AWS services and VPC endpoint services powered by PrivateLink without requiring an internet gateway, NAT device, VPN connection, or AWS Direct Connect connection. Instances in your VPC do not require public IP addresses to communicate with resources in the service. Traffic between your VPC and the other service does not leave the Amazon network.

### IP addressing

* Once the VPC is created, its CIDR block range can NOT be chagned.
* To change CIDR size, you need to create a new VPC
* The different subnets within a VPC can NOT be overlap.
* Can expand VPC by adding secondary IPv4 CIDR blocks

### Default VPC 

AWS creates a default VPC for you in each region. The default VPC will include 1 CIDR block, 1 route table, 1 DHCP options set, 1 Network ACL, 1 Security Group, 1 Internet Gateway, and 3~6 Subnets. The number of subnet depends on the number of Available Zone in the region.  

### Subnet


When you create a VPC, you must specify a range of IPv4 addresses for the VPC in the form of a Classless Inter-Domain Routing (CIDR) block; for example, 10.0.0.0/16. This is the primary CIDR block for your VPC. 


#### Public subnet

A subnet's traffic is routed to an internet gateway.

#### Private subnet

A subnet doesn't have a route an internet gateway.

#### VPN-only subnet

A subnet has traffic routed to a virtual private gateway for a Site-to-Site VPN connection.

#### Reserved IPs

In each subnet CIDR block are not available for you to use, and cannot be assigned to an instance. For example, in a subnet with CIDR block 10.0.1.0/24, the following five IP addresses are reserved:

* 10.0.1.0: Network address.

* 10.0.1.1: Reserved by AWS for the VPC router.

* 10.0.1.2: Reserved by AWS.DNS related. The IP address of the DNS server is the base of the VPC network range plus two. For VPCs with multiple CIDR blocks, the IP address of the DNS server is located in the primary CIDR. We also reserve the base of each subnet range plus two for all CIDR blocks in the VPC. For more information, see Amazon DNS Server.

* 10.0.1.3: Reserved by AWS for future use.

* 10.0.1.255: Network broadcast address. We do not support broadcast in a VPC, therefore we reserve this address.


### Internet Gateway

* It is the gateway through which your VPC comminuicates with the internet, and with other AWS services
* It is horizontally scaled, redundant, and highly available VPC component
* It performant NAT (static one-to-one) between your Private and Public (or Elastic) IPv4 addresses
* Every VPC can have only one Internet Gateway

### VPN Connection

VPN connection links your data center (or network) to your Amazon Virtual Private Cloud (VPC). A customer gateway device is the anchor on your side of that connection. It can be a physical or software appliance. The anchor on the AWS side of the VPN connection is called a virtual private gateway.



<file_sep>+++
title = "Error handling"
description="Rustlang Introduction: Error "
weight = 3
+++


### Error

* Rust groups errors into two major categories: recoverable and unrecoverable errors. 

* Rust doesn’t have exceptions. Instead, it has the type Result<T, E> for recoverable errors and the panic! macro that stops execution when the program encounters an unrecoverable error. 


#### panic! - Unrecoverable Errors

* Rust has the panic! macro. When the panic! macro executes, your program will print a failure message, unwind and clean up the stack, and then quit. This most commonly occurs when a bug of some kind has been detected and it’s not clear to the programmer how to handle the error.


#### Unwinding the Stack or Aborting in Response

* By default, when a panic occurs, the program starts unwinding, which means Rust walks back up the stack and cleans up the data from each function it encounters. But this walking back and cleanup is a lot of work. The alternative is to immediately abort, which ends the program without cleaning up. Memory that the program was using will then need to be cleaned up by the operating system. 

* Abort on panic in release mode 

```toml
[profile.release]
panic = 'abort'
```

##### Using a panic! Backtrace

* panic! call comes from a library because of a bug in our code instead of from our code calling the macro directly.

* Backtraces in Rust work as they do in other languages: the key to reading the backtrace is to start from the top and read until you see files you wrote. 


#### Result  - Recoverable Errors


* Most errors aren’t serious enough to require the program to stop entirely.

* Result enum is defined as having two variants, Ok and Err

    ```rs
    enum Result<T, E> {
        Ok(T),
        Err(E),
    }
    ```

* Matching on Different Errors


    ```rs
    use std::fs::File;
    use std::io::ErrorKind;

    fn main() {
        let f = File::open("hello.txt");

        let f = match f {
            Ok(file) => file,
            Err(error) => match error.kind() {
                ErrorKind::NotFound => match File::create("hello.txt") {
                    Ok(fc) => fc,
                    Err(e) => panic!("Problem creating the file: {:?}", e),
                },
                other_error => panic!("Problem opening the file: {:?}", other_error),
            },
        };
    }
    ```

#### Shortcuts for Panic on Error: unwrap and expect

*  If the Result value is the Ok variant, unwrap will return the value inside the Ok. If the Result is the Err variant, unwrap will call the panic! macro for us. 

* Another method, expect, which is similar to unwrap, lets us also choose the panic! error message. Using expect instead of unwrap and providing good error messages can convey your intent and make tracking down the source of a panic easier.

    ```rs
    use std::fs::File;

    fn main() {
        let f = File::open("hello.txt").unwrap();
        let f = File::open("hello.txt")
                    .expect("Failed to open hello.txt");
    }
    ```

#### Propagating Errors

* Propagating the error gives more control to the calling code, where there might be more information or logic that dictates how the error should be handled than what you have available in the context of your code.

* Example of reading file content to string

    ```rs
    use std::io;
    use std::io::Read;
    use std::fs::File;
    fn read_username_from_file() -> Result<String, io::Error> {
        let f = File::open("hello.txt");

        let mut f = match f {
            Ok(file) => file,
            Err(e) => return Err(e),
        };

        let mut s = String::new();

        match f.read_to_string(&mut s) {
            Ok(_) => Ok(s),
            Err(e) => Err(e),
        }
    }
    ```

* This pattern of propagating errors is so common in Rust that Rust provides the question mark operator ? to make this easier.

* A Shortcut for Propagating Errors: the ? Operator

    ```rs
    use std::io;
    use std::io::Read;
    use std::fs::File;
    fn read_username_from_file() -> Result<String, io::Error> {
        let mut f = File::open("hello.txt")?;
        let mut s = String::new();
        f.read_to_string(&mut s)?;
        Ok(s)
    }
    ```

* A more concise sample 

    ```rs
    use std::io;
    use std::io::Read;
    use std::fs::File;

    fn read_username_from_file() -> Result<String, io::Error> {
        let mut s = String::new();

        File::open("hello.txt")?.read_to_string(&mut s)?;

        Ok(s)
    }
    ```

##### The ? Operator Can Only Be Used in Functions

* The ? operator can only be used in functions that have a return type of Result, because it is defined to work in the same way as the match expression 


#### To panic! or Not to panic!

__When code panics, there’s no way to recover. You could call panic! for any error situation, whether there’s a possible way to recover or not, but then you’re making the decision on behalf of the code calling your code that a situation is unrecoverable. When you choose to return a Result value, you give the calling code options rather than making the decision for it. The calling code could choose to attempt to recover in a way that’s appropriate for its situation, or it could decide that an Err value in this case is unrecoverable, so it can call panic! and turn your recoverable error into an unrecoverable one.__

* Similarly, the unwrap and expect methods are very handy when prototyping, before you’re ready to decide how to handle errors. They leave clear markers in your code for when you’re ready to make your program more robust.

* If a method call fails in a test, you’d want the whole test to fail, even if that method isn’t the functionality under test. Because panic! is how a test is marked as a failure, calling unwrap or expect is exactly what should happen.


#### Guidelines for Error Handling

* The bad state is not something that’s expected to happen occasionally.

* Your code after this point needs to rely on not being in this bad state.
    
* There’s not a good way to encode this information in the types you use.

* When failure is expected, it’s more appropriate to return a Result than to make a panic! 


#### Custom Types for Validation

* Sample of validation with loop

    ```rs
    loop {
        // --snip--

        let guess: i32 = match guess.trim().parse() {
            Ok(num) => num,
            Err(_) => continue,
        };

        if guess < 1 || guess > 100 {
            println!("The secret number will be between 1 and 100.");
            continue;
        }

        match guess.cmp(&secret_number) {
        // --snip--
    }
    ```

* Above  is not an ideal solution.  it’s safe for functions to use the new type in their signatures and confidently use the values they receive. 

    ```rs
    pub struct Guess {
        value: i32,
    }

    impl Guess {
        pub fn new(value: i32) -> Guess {
            if value < 1 || value > 100 {
                panic!("Guess value must be between 1 and 100, got {}.", value);
            }

            Guess {
                value
            }
        }

        pub fn value(&self) -> i32 {
            self.value
        }
    }
    ```

*  The panic! macro signals that your program is in a state it can’t handle and lets you tell the process to stop instead of trying to proceed with invalid or incorrect values. The Result enum uses Rust’s type system to indicate that operations might fail in a way that your code could recover from. 

























<file_sep>+++
title = "F# Functions"
description = "F# Function, Rec Func & Inline Func"
weight = 3
+++


## F# Function
> Functions are the fundamental unit of program execution in any programming language. As in other languages, an F# function has a name, can have parameters and take arguments, and has a body. F# also supports functional programming constructs such as treating functions as values, using unnamed functions in expressions, composition of functions to form new functions, curried functions, and the implicit definition of functions by way of the partial application of function arguments.

- Syntax

```fsharp
// Non-recursive function definition.
let [inline] function-name parameter-list [ : return-type ] = function-body
// Recursive function definition.
let rec function-name parameter-list = recursive-function-body
```

### F# Recursive Function 

> The rec keyword is used together with the let keyword to define a recursive function.

- Example - Bad implementation

```fsharp 
let rec fib n =
    match n with
    | 0 | 1 -> n
    | n -> fib (n-1) + fib (n-2)
```

- Example - Good one: Tail Recursiot

Tail recursion is important because it can be implemented more efficiently than general recursion. When we make a normal recursive call, we have to push the return address onto the call stack then jump to the called function. This means that we need a call stack whose size is linear in the depth of the recursive calls. When we have tail recursion we know that as soon as we return from the recursive call we're going to immediately return as well, so we can skip the entire chain of recursive functions returning and return straight to the original caller. That means we don't need a call stack at all for all of the recursive calls, and can implement the final call as a simple jump, which saves us space.

```fsharp
let fib n =
    let rec loop acc1 acc2 n =
        match n with
        | 0 -> acc1
        | 1 -> acc2
        | _ ->
            loop acc2 (acc1 + acc2) (n - 1)
    loop 0 1 n
```

### F# Inline Function

> Inline functions are functions that are integrated directly into the calling code.

When you use static type parameters, any functions that are parameterized by type parameters must be inline. This guarantees that the compiler can resolve these type parameters. 

You should avoid using inline functions for optimization unless you have tried all other optimization techniques.


 
```fsharp
// The following code example illustrates an inline function at the top level, an inline instance method, and an inline static method.
let inline increment x = x + 1
type WrapInt32() =
    member inline this.incrementByOne(x) = x + 1
    static member inline Increment(x) = x + 1
```



<file_sep>+++
title = "MySql: Getting Started"
description="Start / Stop MySql, Reset root credential"
+++

> MySQL is the world's most popular open source database. Whether you are a fast growing web property, technology ISV or large enterprise, MySQL can cost-effectively help you deliver high performance, scalable database applications.

### Prerequisite

* Install MySql 5.6+ on your PC or server 



### Launch & Stop MySql server

```bash
# use systemctl
systemctl status mysql.service
systemctl restart mysql.service

# use service
service mysql status
service mysql restart 

```

### Set root password

```bash
mysql_secure_installation
```

### Reset root password

```sql
-- MySql 5.6.x
mysql> UPDATE mysql.user SET password = PASSWORD('<PASSWORD>') 
    -> WHERE User = 'root' AND Host = 'localhost';

-- MySql 5.7+
mysql> UPDATE mysql.user SET authentication_string = PASSWORD('<PASSWORD>') 
    -> WHERE User = 'root' AND Host = 'localhost'; 

mysql> FLUSH PRIVILEGES;

```


### Connect to MySql server

```bash
mysql -u <user_name> -p  -P <port> -h <host_name>
```


### Add a new user

```sql
CREATE USER 'user1'@'%' IDENTIFIED BY '<PASSWORD>';
GRANT ALL PRIVILEGES ON * . * TO 'user1'@'%';
```


### Delete a user

```sql
REVOKE ALL PRIVILEGES, GRANT OPTION FROM 'user1'@'%';
DROP USER 'user1'@'%';
```
<file_sep>+++
title = "Management"
weight = 4
+++

{{%children style="card" description="true" sort="Weight" %}}<file_sep>+++
title = "AWS : CLI - 2"
description = "AWS CLI & VPC"
weight=2
+++


### AWS CLI & VPC

Following is a sample to create a VPC with 2 private subnets, 2 public subnets across 2 avaliable zones and NAT Gateway. 

```bash
#!/bin/bash
#******************************************************************************
#    AWS VPC CLI Script
#******************************************************************************
#
# SYNOPSIS
#    Automates the creation of a custom IPv4 VPC, having both a public and a
#    private subnet, and a NAT gateway.
#
#==============================================================================
#
# NOTES
#   VERSION:   1.0
#   AUTHOR:    <NAME>
#
#==============================================================================
#   MODIFY THE SETTINGS BELOW
#==============================================================================
#
AWS_REGION="ap-southeast-2"
VPC_NAME="DEV-PG-II"
VPC_CIDR="10.5.0.0/16"
SUBNET_PUBLIC_CIDR="10.5.1.0/24"
SUBNET_PUBLIC_AZ="ap-southeast-2a"
SUBNET_PUBLIC_NAME="$VPC_NAME-PubSub-AZ2a"
SUBNET_PRIVATE_CIDR="10.5.2.0/24"
SUBNET_PRIVATE_AZ="ap-southeast-2b"
SUBNET_PRIVATE_NAME="$VPC_NAME-PrvSub-AZ2b"
IGW_NAME="$VPC_NAME-IGW"
NAT_GW_NAME="$VPC_NAME-NAT-GW"
CHECK_FREQUENCY=5
#
#==============================================================================
#   DO NOT MODIFY CODE BELOW
#==============================================================================
#
# Create VPC
echo "Creating VPC in preferred region..."
VPC_ID=$(aws ec2 create-vpc \
  --cidr-block $VPC_CIDR \
  --query 'Vpc.{VpcId:VpcId}' \
  --output text \
  --region $AWS_REGION)
echo "  VPC ID '$VPC_ID' CREATED in '$AWS_REGION' region."

# Add Name tag to VPC
aws ec2 create-tags \
  --resources $VPC_ID \
  --tags "Key=Name,Value=$VPC_NAME" \
  --region $AWS_REGION
echo "  VPC ID '$VPC_ID' NAMED as '$VPC_NAME'."

# Create Public Subnet
echo "Creating Public Subnet..."
SUBNET_PUBLIC_ID=$(aws ec2 create-subnet \
  --vpc-id $VPC_ID \
  --cidr-block $SUBNET_PUBLIC_CIDR \
  --availability-zone $SUBNET_PUBLIC_AZ \
  --query 'Subnet.{SubnetId:SubnetId}' \
  --output text \
  --region $AWS_REGION)
echo "  Subnet ID '$SUBNET_PUBLIC_ID' CREATED in '$SUBNET_PUBLIC_AZ'" \
  "Availability Zone."

# Add Name tag to Public Subnet
aws ec2 create-tags \
  --resources $SUBNET_PUBLIC_ID \
  --tags "Key=Name,Value=$SUBNET_PUBLIC_NAME" \
  --region $AWS_REGION
echo "  Subnet ID '$SUBNET_PUBLIC_ID' NAMED as" \
  "'$SUBNET_PUBLIC_NAME'."

# Create Private Subnet
echo "Creating Private Subnet..."
SUBNET_PRIVATE_ID=$(aws ec2 create-subnet \
  --vpc-id $VPC_ID \
  --cidr-block $SUBNET_PRIVATE_CIDR \
  --availability-zone $SUBNET_PRIVATE_AZ \
  --query 'Subnet.{SubnetId:SubnetId}' \
  --output text \
  --region $AWS_REGION)
echo "  Subnet ID '$SUBNET_PRIVATE_ID' CREATED in '$SUBNET_PRIVATE_AZ'" \
  "Availability Zone."

# Add Name tag to Private Subnet
aws ec2 create-tags \
  --resources $SUBNET_PRIVATE_ID \
  --tags "Key=Name,Value=$SUBNET_PRIVATE_NAME" \
  --region $AWS_REGION
echo "  Subnet ID '$SUBNET_PRIVATE_ID' NAMED as '$SUBNET_PRIVATE_NAME'."

# Create Internet gateway
echo "Creating Internet Gateway..."
IGW_ID=$(aws ec2 create-internet-gateway \
  --query 'InternetGateway.{InternetGatewayId:InternetGatewayId}' \
  --output text \
  --region $AWS_REGION)
echo "  Internet Gateway ID '$IGW_ID' CREATED."

# Add Name tag to Internet gateway
aws ec2 create-tags \
  --resources $IGW_ID \
  --tags "Key=Name,Value=$IGW_NAME" \
  --region $AWS_REGION
echo " Internet gateway '$IGW_ID' NAMED as '$IGW_NAME'."


# Attach Internet gateway to your VPC
aws ec2 attach-internet-gateway \
  --vpc-id $VPC_ID \
  --internet-gateway-id $IGW_ID \
  --region $AWS_REGION
echo "  Internet Gateway ID '$IGW_ID' ATTACHED to VPC ID '$VPC_ID'."

# Create Route Table
echo "Creating Route Table..."
ROUTE_TABLE_ID=$(aws ec2 create-route-table \
  --vpc-id $VPC_ID \
  --query 'RouteTable.{RouteTableId:RouteTableId}' \
  --output text \
  --region $AWS_REGION)
echo "  Route Table ID '$ROUTE_TABLE_ID' CREATED."


# Create route to Internet Gateway
RESULT=$(aws ec2 create-route \
  --route-table-id $ROUTE_TABLE_ID \
  --destination-cidr-block 0.0.0.0/0 \
  --gateway-id $IGW_ID \
  --region $AWS_REGION)
echo "  Route to '0.0.0.0/0' via Internet Gateway ID '$IGW_ID' ADDED to" \
  "Route Table ID '$ROUTE_TABLE_ID'."

# Associate Public Subnet with Route Table
RESULT=$(aws ec2 associate-route-table  \
  --subnet-id $SUBNET_PUBLIC_ID \
  --route-table-id $ROUTE_TABLE_ID \
  --region $AWS_REGION)
echo "  Public Subnet ID '$SUBNET_PUBLIC_ID' ASSOCIATED with Route Table ID" \
  "'$ROUTE_TABLE_ID'."

# Enable Auto-assign Public IP on Public Subnet
aws ec2 modify-subnet-attribute \
  --subnet-id $SUBNET_PUBLIC_ID \
  --map-public-ip-on-launch \
  --region $AWS_REGION
echo "  'Auto-assign Public IP' ENABLED on Public Subnet ID" \
  "'$SUBNET_PUBLIC_ID'."

# Allocate Elastic IP Address for NAT Gateway
echo "Creating NAT Gateway..."
EIP_ALLOC_ID=$(aws ec2 allocate-address \
  --domain vpc \
  --query '{AllocationId:AllocationId}' \
  --output text \
  --region $AWS_REGION)
echo "  Elastic IP address ID '$EIP_ALLOC_ID' ALLOCATED."

# Create NAT Gateway
NAT_GW_ID=$(aws ec2 create-nat-gateway \
  --subnet-id $SUBNET_PUBLIC_ID \
  --allocation-id $EIP_ALLOC_ID \
  --query 'NatGateway.{NatGatewayId:NatGatewayId}' \
  --output text \
  --region $AWS_REGION)

FORMATTED_MSG="Creating NAT Gateway ID '$NAT_GW_ID' and waiting for it to "
FORMATTED_MSG+="become available.\n    Please BE PATIENT as this can take some "
FORMATTED_MSG+="time to complete.\n    ......\n"
printf "  $FORMATTED_MSG"
FORMATTED_MSG="STATUS: AVAILABLE - Total of %02d seconds elapsed for process"
FORMATTED_MSG+="\n    ......\n  NAT Gateway ID '%s' is now AVAILABLE.\n"
start_time="$(date -u +%s)"

aws ec2 wait nat-gateway-available \
  --nat-gateway-ids $NAT_GW_ID

end_time="$(date -u +%s)"
elapsed="$(($end_time-$start_time))" 

printf "  $FORMATTED_MSG" $elapsed $NAT_GW_ID

# Add Name tag to NAT Gateway
aws ec2 create-tags \
  --resources $NAT_GW_ID \
  --tags "Key=Name,Value=$NAT_GW_NAME" \
  --region $AWS_REGION
echo " Internet gateway '$NAT_GW_ID' NAMED as '$NAT_GW_NAME'."


# Create route to NAT Gateway
MAIN_ROUTE_TABLE_ID=$(aws ec2 describe-route-tables \
  --filters Name=vpc-id,Values=$VPC_ID Name=association.main,Values=true \
  --query 'RouteTables[*].{RouteTableId:RouteTableId}' \
  --output text \
  --region $AWS_REGION)
echo "  Main Route Table ID is '$MAIN_ROUTE_TABLE_ID'."
RESULT=$(aws ec2 create-route \
  --route-table-id $MAIN_ROUTE_TABLE_ID \
  --destination-cidr-block 0.0.0.0/0 \
  --gateway-id $NAT_GW_ID \
  --region $AWS_REGION)
echo "  Route to '0.0.0.0/0' via NAT Gateway with ID '$NAT_GW_ID' ADDED to" \
  "Route Table ID '$MAIN_ROUTE_TABLE_ID'."
echo "COMPLETED"

```









<file_sep>+++
date = "2016-12-07T16:56:21+11:00"
title = "Tensorflow Note - 2"
description = "Tensorflow Note - 2"

+++


## Prerequisites

* You are using Windows 7 or higher version
* You are using Anaconda to setup the environment


## Install Anaconda 

> Anaconda® is a package manager, an environment manager, a Python/R data science distribution, and a collection of over 1,500+ open source packages. Anaconda is free and easy to install, and it offers free community support.

* [Please download Anaconda from the official site](https://www.continuum.io/downloads)
  
    

## Create tensorflow virtualenv with python 3.5
* Anaconda uses python 3.6 by default. Tensorflow only supports python 3.5. 

    cd /path/to/envs
    conda create -n tensorflow
    

## Install tensorflow 

    activate tensorflow
    ## For CPU
    pip install --ignore-installed --upgrade \
        https://storage.googleapis.com/tensorflow/windows/cpu/tensorflow-1.1.0-cp35-cp35m-win_amd64.whl

    ## Or for GPU
    pip install --ignore-installed --upgrade \
        https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-1.1.0-cp35-cp35m-win_amd64.whl


## Use sample code to test Tensorflow

### Save code below to file test.py 

```python
import numpy as np
import tensorflow as tf

## Model parameters
W = tf.Variable([.3], tf.float32)
b = tf.Variable([-.3], tf.float32)
## Model input and output
x = tf.placeholder(tf.float32)
linear_model = W * x + b
y = tf.placeholder(tf.float32)
## loss
loss = tf.reduce_sum(tf.square(linear_model - y)) ## sum of the squares
## optimizer
optimizer = tf.train.GradientDescentOptimizer(0.01)
train = optimizer.minimize(loss)
## training data
x_train = [1,2,3,4]
y_train = [0,-1,-2,-3]
## training loop
init = tf.global_variables_initializer()
sess = tf.Session()
sess.run(init) ## reset values to wrong
for i in range(1000):
sess.run(train, {x:x_train, y:y_train})

## evaluate training accuracy
curr_W, curr_b, curr_loss  = sess.run([W, b, loss], {x:x_train, y:y_train})
print("W: %s b: %s loss: %s"%(curr_W, curr_b, curr_loss))
```

### Test with tensorflow-gpu (GPU enabled)

    activate tensorflow
    cd /ws/python/tf
    python3 test.py

    ## You will probably see the result as follow 
    ## ....
    ## name: GeForce GTX 850M
    ## major: 5 minor: 0 memoryClockRate (GHz) 0.9015
    ## pciBusID 0000:0a:00.0
    ## Total memory: 3.95GiB
    ## Free memory: 3.58GiB
    ## 2017-04-25 10:25:59.640621: I tensorflow/core/common_runtime/gpu/gpu_device.cc:908] DMA: 0 
    ## 2017-04-25 10:25:59.640626: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 0:   Y 
    ## 2017-04-25 10:25:59.640640: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] 
    ##    Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 850M, pci ## bus id: 0000:0a:00.0)
    ## W: [-0.9999969] b: [ 0.99999082] loss: 5.69997e-11










<file_sep>+++
title = "F# Async"
description = "F# Async expressions"
weight = 12
+++


## Async expressions

Async expressions provide one way of performing computations asynchronously, that is, without blocking execution of other work.


### Asynchronous Binding by Using let!

The effect of let! is to enable execution to continue on other computations or threads as the computation is being performed. After the right side of the let! binding returns, the rest of the async expression resumes execution.

```fsharp
// let just stores the result as an asynchronous operation.
let (result1 : Async<byte[]>) = stream.AsyncRead(bufferSize)
// let! completes the asynchronous operation and returns the data.
let! (result2 : byte[])  = stream.AsyncRead(bufferSize)
```

`let!` can only be used to await F# async computations `Async<T>` directly. You can await other kinds of asynchronous operations indirectly:

- .NET tasks, `Task<TResult>` and the non-generic Task, by combining with `Async.AwaitTask`
- .NET value tasks, `ValueTask<TResult>` and the non-generic ValueTask, by combining with `.AsTask()` and `Async.AwaitTask`
- Any object following the "GetAwaiter" pattern specified in F# RFC FS-1097, by combining with `task { return! expr } |> Async.AwaitTask`.


###  use and use! bindings

In addition to `let!`, you can use use! to perform asynchronous bindings. The difference between let! and use! is the same as the difference between `let` and `use`. For `use!,` the object is disposed of at the close of the current scope. Note that in the current release of F#, `use!` does not allow a value to be initialized to null, even though use does.

### Asynchronous Primitives

A method that performs a single asynchronous task and returns the result is called an asynchronous primitive, and these are designed specifically for use with let!. Several asynchronous primitives are defined in the F# core library. 

You use the function `Async.RunSynchronously` to execute an asynchronous operation and wait for its result. As an example, you can execute multiple asynchronous operations in parallel by using the Async.Parallel function together with the `Async.RunSynchronously` function. The `Async.Parallel` function takes a list of the Async objects, sets up the code for each Async task object to run in parallel, and returns an Async object that represents the parallel computation. Just as for a single operation, you call `Async.RunSynchronously` to start the execution.


```fsharp

open System.Net
open Microsoft.FSharp.Control.WebExtensions

let urlList = [ "Microsoft.com", "http://www.microsoft.com/"
                "MSDN", "http://msdn.microsoft.com/"
                "Bing", "http://www.bing.com"
              ]

let fetchAsync(name, url:string) =
    async {
        try
            let uri = new System.Uri(url)
            let webClient = new WebClient()
            let! html = webClient.AsyncDownloadString(uri)
            printfn "Read %d characters for %s" html.Length name
        with
            | ex -> printfn "%s" (ex.Message);
    }

let runAll() =
    urlList
    |> Seq.map fetchAsync
    |> Async.Parallel
    |> Async.RunSynchronously
    |> ignore

runAll()
```

<file_sep>#!/bin/bash


export SITE_URL="www.us2.io"
export SITE_SSL_PORT="443"

export SITE_URLS=(
    www.applydirect.com.au
    www.applydirect.co.nz
    careersvic.applydirect.com.au
    iworkfornsw.applydirect.com.au
    live-iworkfornsw.applydirect.com.au
    www.ad1holdings.com.au
    mypharmacycareer.com.au
    core-blue.us2.io
    esi.uss.technology
    www.utilitysoftwareservices.com
    www.utilitiessoftwareservices.com
    www.arcenergygroup1.com.au
    www.planenergy1.com.au
    nextconnectnbe.com.au
    ws-proxy.us2.uss
    uss.igeno.com.au
)

DOMAIN_LIST="domain_list.csv"

echo "Domain, Issuer, Start Date, End Date" > ${DOMAIN_LIST}

for SITE_URL in "${SITE_URLS[@]}"; do
   echo $SITE_URL
   data=$(true | openssl s_client -connect ${SITE_URL}:${SITE_SSL_PORT} \
         -servername ${SITE_URL} 2> /dev/null |  \
        openssl x509 -text   | grep "Issuer:" -A 3 ) # -issuer -email -startdate -enddate

    if [[ ! -z $data ]]; then
        data=${data/"Issuer:"/"$SITE_URL,\""}
        data=${data/"Not Before:"/"\","}
        data=${data/"Not After :"/","}
        echo $data >> ${DOMAIN_LIST}
    else 
        echo "$SITE_URL,N/A,N/A,N/A">> ${DOMAIN_LIST}
    fi;
    sleep 5
    
done;

cat $DOMAIN_LIST<file_sep>+++
title = "Adv Bash - 1"
description = "Reference Cards - Special Characters & Operators"
+++

### Special Characters

What makes a character special? If it has a meaning beyond its literal meaning, a meta-meaning, then we refer to it as a special character. Along with commands and keywords, special characters are building blocks of Bash scripts.


#### Special Shell Variables


  Variable |	Meaning 
|--------|--------|
$0	     |   Filename of script
$1	     |   Positional parameter #1
$2 - $9	 |   Positional parameters #2 - #9
${10}	 |   Positional parameter #10
$#	     |   Number of positional parameters
"$*"	 |   All the positional parameters (as a single word) *
"$@"	 |   All the positional parameters (as separate strings)
${#*}	 |   Number of positional parameters
${#@}	 |   Number of positional parameters
$?	     |   Return value
$$	     |   Process ID (PID) of script
$-	     |   Flags passed to script (using set)
$_	     |   Last argument of previous command
$!	     |   Process ID (PID) of last job run in background


#### Operator Precedence

In a script, operations execute in order of precedence: the higher precedence operations execute before the lower precedence ones.



 Operator | Meaning | Comments
|----|----|------
|| HIGHEST PRECEDENCE
|var++ var--	|post-increment, post-decrement	|C-style operators
| ++var --var	|pre-increment, pre-decrement	 
| ! ~	|negation|	logical / bitwise, inverts sense of following operator 
| **	|exponentiation|	arithmetic operation
| * / %	|multiplication, division, modulo|	arithmetic operation
| + -	|addition, subtraction|	arithmetic operation
| << >>	|left, right| shift	bitwise
| -z -n	|unary comparison|	string is/is-not null
| -e -f -t -x, etc.	|unary comparison|	file-test
| < -lt > -gt <= -le >= -ge	|compound comparison|	string and integer
| -nt -ot -ef	|compound comparison|	file-test
| == -eq != -ne|equality / inequality |	test operators, string and integer
| &	    |AND	|bitwise
| ^	    |XOR	|exclusive OR, bitwise
| \|	    |OR	|bitwise
| && -a	|AND|	logical, compound comparison
| \| -o	|OR |logical, compound comparison 
| ?:	|trinary operator	|C-style
| =	|assignment|	(do not confuse with equality test)
| *= /= %= += -= <<= >>= &=	|combination assignment|	times-equal, divide-equal,|  mod-equal, etc.
| ,	|comma|	links a sequence of operations
|||LOWEST PRECEDENCE


####  TEST Operators: Binary Comparison

Operator|Meaning|Operator|Meaning
----|----|----|---- 
Arithmetic Comparison|	 	 	|String Comparison|	 
-eq	|Equal to| = |	Equal to
||| == | 	Equal to
-ne	| Not equal to	 |	!=	| Not equal to
-lt	| Less than	 	\<	Less than (ASCII) *
-le	| Less than or equal to	 | | 	 	 
-gt	| Greater than	 |	\>	| Greater than (ASCII) *
-ge	| Greater than or equal to | | |	 	 	 
||| -z |	String is empty
||| -n |    String is not empty
Arithmetic Comparison| within double parentheses (( ... ))	 | | 	 	 
>	|Greater than	 	 	 
>=	|Greater than or equal to	 	 	 
<	|Less than	 	 	 
<=	|Less than or equal to	 	 	 


<file_sep>+++
title = "JS & ES Note - 1"
description="The most unpredictable [this] in JavaScript"

+++


## The most unpredictable keyword and feature

You may already guessed what I am talking about. Yes,  the**this** keyword. It is not only a powerful feature, but also often misinterpreted keyword.

In JavaScript, we also have this concept inside a Function constructor when it is invoked using the “new” keyword, however it is not the only rule and “this” can often refer to a different object from a different execution context. 


### **this** can be anything
 
* **this** represents different things. 


  ```

  var a = "abc"
  console.log(a);

  this.a = "123";
  console.log(a);

  console.log(this);

  ```

- Do you know what output will be?

  - Running in the console of Chrome / Firefox

```
abc
123
Window {postMessage: ƒ, blur: ƒ, focus: ƒ, close: ƒ, parent: Window, …}
```

  - Running in the js file via node 

  ```
  # Save the content as index.js
  # Run command - node index.js

  abc
  abc
  { a: '123' }
  ```
    
  - Running in the node command prompt

  ```
  abc
  123
  Object [global] {global: [Circular],...}
  ```

* Basically the **this** represents the context environment, AKA, the global object. The best way to confirm what the **this** means is better to print it out. 


### **this** inside a function

From 2015, ECMAScript 6, aka ES 2015, has been supported by most modern browsers. So when we talk about function in JavaScript, literally we are talking two slightly different functions. One is defined with keyword `function`, another one is aka arrow function or fat arrow. For most scenario, arrow function is the same as old style function, but fat arrow handles the **this** keyword in a more stable and predictable way.  

> Mozilla - MDN: The value of this is determined by how a function is called (runtime binding). It can't be set by assignment during execution, and it may be different each time the function is called. ES5 introduced the bind() method to set the value of a function's this regardless of how it's called. The arrow functions which don't provide their own this binding (it retains the this value of the enclosing lexical context).

According MDN's explanation, the arrow function is a better implementation. It is more predictable and stable. Arrow function is highly recommended to replace the old style function.


#### **this** inside a regular function 

* The samples below show the different *this*es

  ```js
  var obj0 = {
      prop: 'obj0',
      func_1: function () {
          console.log(' func_1-> ', this);
          var func_2 = {
              prop: 'func_2',
              func_3: function () {
                  console.log(' func_3 -> ', this);
              }
          };
          func_2.func_3();
      }
  };
  obj0.func_1();
  ////////////////////////////////
  var obj1 = {
      prop: 'obj1',
      func_1: function () {
          console.log(' func_1-> ', this);
          func_2 = function () {
              this.prop = 'func_2';
              var func_3 = function () {
                  this.p = 'func_3';
                  console.log(' func_3 -> ', this);
              }
              return func_3
          };
          func_2()();
      }
  };
  obj1.func_1();
  ```


* Test (Fat Arrow)

```js
var obj2 = {
    prop: 123,
    func_1: () => {
        console.log(' func_1-> ', this);
        func_2 = {
            prop: 'func_2',
            func_3: () => {
                this.p = 'func_3';
                console.log(' func_3 -> ', this);
            }
        };
        func_2.func_3();
    }
};
obj2.func_1();
```



#### Fat arrow has no binding

* The sample below show the different binding behavior between regular function and arrow function 


  ```js
  var obj3 = {
      p: 'obj3',
      toBeCalled: function() {
        console.log(' this is toBeCalled ', this.p);
      },
      toBind: function(obj) {
        obj.toBeCalled();
      }
  };

  var testBind = obj3.toBind;
  testBind(obj3);

  /// output:
  ///  this is toBeCalled  obj3

  var obj4 = {
      p: 'obj4',
      toBeCalled: () => {
        console.log(' this is toBeCalled ', this.p);
      },
      toBind: obj => {
        obj.toBeCalled();
      }
  };

  var testBind2 = obj4.toBind;
  testBind2(obj4);

  /// output
  ///  this is toBeCalled  undefined

  ```

<file_sep>+++
title = "Azure: CAF - 3"
weight = 1
description="Strategy of CAF"
+++

## cloud adoption strategy

- Define and document your motivations: Meet with key stakeholders and executives to document the motivations behind cloud adoption.
	
- Document business outcomes: Engage motivated stakeholders and executives to document specific business outcomes.
	
- Develop a business case: Develop a business case to validate the financial model that supports your motivations and outcomes.
	
- Choose the right first project: Your first cloud adoption project will help align motivations with technical effort. This article can help you choose your first project wisely.


### Motivation-driven strategies

Business transformations that are supported by cloud adoption can be driven by various motivations. 


#### Migration

- Cost savings.
- Reduction in vendor or technical complexity.
- Optimization of internal operations.
- Increasing business agility.
- Preparing for new technical capabilities.
- Scaling to meet market demands.
- Scaling to meet geographic demands.

#### Innovation

- Increasing business agility.
- Preparing for new technical capabilities.
- Building new technical capabilities.
- Scaling to meet market demands.
- Scaling to meet geographic demands.
- Improving customer experiences and engagements.
- Transforming products or services.



### Business Outcomes


* Fiscal outcomes -Financial or fiscal performance is the cleanest business outcome for many business leaders, but not the only one.

* Agility outcomes - Today's fast-changing business environment places a premium on time. The ability to respond to and drive market change quickly is the fundamental measure of business agility.
  
* Reach outcomes - In a constantly shrinking market, global reach (ability to support global customers and users) can be measured by compliance in geographies that are relevant to the business.

* Customer engagement outcomes - Social marketplaces are redefining winners and losers at an unheard-of pace. Responding to user needs is a key measure of customer engagement.

* Performance outcomes - Performance and reliability are assumed. When either falters, reputation damage can be painful and long-lasting.


### Business Justification

On a basic level, the business justification focuses on the return on investment (ROI) associated with the proposed technical change. The generic formula for ROI is:

> ROI (Return of Investment) = (Gain from Investment -  Investment) /  Investment

When the ROI is below 20%, consider a digital estate planning exercise, paying specific attention to rationalization. 

Calculating the gain from investment often requires a second formula that's specific to the business outcomes and associated technical changes. Calculating earnings is harder than calculating cost reductions.

> Gain from Investment = Revenue deltas - Cost deltas

#### Revenue deltas

Revenue deltas should be forecast in partnership with business stakeholders. 

#### Cost deltas

Cost deltas are the amount of increase or decrease that will be caused by the transformation. 










<file_sep>+++
date = "2016-12-07T16:56:21+11:00"
title = "CNTK Note - 1"
description="AI Framework from Microsoft"

+++


##  Prerequisites

* You are using Windows 7 or higher version
* You are using Anaconda to setup the environment

##   Create CNTK virtual environment

###   use follow command to remove existing virtual environment
    
        conda remove -n cntk --all
        conda create -n cntk 

###  Activate virtual environment and install CNTK

    activate cntk
    pip install https://cntk.ai/PythonWheel/CPU-Only/cntk-2.0rc3-cp36-cp36m-win_amd64.whl

##  Test CNTK

    python
    >>> import cntk
    >>> cntk.__version__
    '2.0rc3'



<file_sep>+++
date = "2018-12-04T14:59:31+11:00"
title = "VPN VyOS setup"
description = "VPN VyOS setup"
+++


## VPN VyOS


VyOS is a fully open source network OS that runs on a wide range of hardware, virtual machines, and cloud providers and offers features for any networks, small and large.

### VyOS on AWS

#### Setup VyOS

* Launch instance with community AMI - VyOS (HVM) 1.x.x

* Customize the setup script 

```bash
#!/bin/bash
source /opt/vyatta/etc/functions/script-template

AWS_PRIVATE_IP=10.104.16.128
AWS_PUBLIC_IP=172.16.31.10
AWS_NAT_SUBNET=10.104.0.0/16
REMOTE_NAT_IP=127.17.12.172
REMOTE_VPN_SUBNET=172.16.58.3/24
REMOTE_1ST_VPN_IP=192.168.127.12
# REMOTE_2ND_VPN_IP=192.168.3.11 # redundant connection not currently used
REMOTE_PRE_SHARED_KEY=Your_Remote_Key

# begin configuration
configure

# input settings using set
set system host-name vyos-vpn

# setting up NAT
set interfaces ethernet eth0 description 'aws-internal'
# create dummy ethernet device to represent REMOTE-provided private IP
set interfaces dummy dum0 address ${REMOTE_NAT_IP}/32
set interfaces dummy dum0 description 'remote-vpn-ip'
# configure SNAT
set nat source rule 100 description 'Internal to REMOTE'
set nat source rule 100 destination address ${REMOTE_VPN_SUBNET}
set nat source rule 100 outbound-interface 'any'
set nat source rule 100 source address ${AWS_NAT_SUBNET}
set nat source rule 100 translation address ${REMOTE_NAT_IP}

# setting up VPN
# set primary ethernet interface as the VPN interface
set vpn ipsec ipsec-interfaces interface 'eth0'
set vpn ipsec nat-traversal 'enable'
set vpn ipsec logging log-modes 'all' 
# esp-group
set vpn ipsec esp-group vpn-nat-esp compression 'disable'
set vpn ipsec esp-group vpn-nat-esp lifetime '28800'
set vpn ipsec esp-group vpn-nat-esp mode 'tunnel'
set vpn ipsec esp-group vpn-nat-esp pfs 'dh-group2'
set vpn ipsec esp-group vpn-nat-esp proposal 1 encryption 'aes256'
set vpn ipsec esp-group vpn-nat-esp proposal 1 hash 'sha1'
# ike-group
set vpn ipsec ike-group vpn-nat-ike ikev2-reauth 'no'
set vpn ipsec ike-group vpn-nat-ike key-exchange 'ikev1'
set vpn ipsec ike-group vpn-nat-ike lifetime '28800'
set vpn ipsec ike-group vpn-nat-ike proposal 1 encryption 'aes256'
set vpn ipsec ike-group vpn-nat-ike proposal 1 hash 'sha512'
set vpn ipsec ike-group vpn-nat-ike proposal 1 dh-group '5'
set vpn ipsec ike-group vpn-nat-ike dead-peer-detection action 'restart'
set vpn ipsec ike-group vpn-nat-ike dead-peer-detection interval '30'
set vpn ipsec ike-group vpn-nat-ike dead-peer-detection timeout '30'
# site-to-site peer
edit vpn ipsec site-to-site peer ${REMOTE_NSW_VPN_IP}
set authentication mode 'pre-shared-secret'
set authentication pre-shared-secret ${REMOTE_PRE_SHARED_KEY} 
set authentication id ${AWS_PUBLIC_IP}
set connection-type 'initiate'
set default-esp-group 'vpn-nat-esp'
set ike-group 'vpn-nat-ike'
set ikev2-reauth 'inherit'
set local-address ${AWS_PRIVATE_IP}
set tunnel 0 local prefix ${REMOTE_NAT_IP}/32
set tunnel 0 remote prefix ${REMOTE_VPN_SUBNET}

# commit command applies changes to VyOS device
commit
# save configuration to machine
save
# exit configuration mode
exit
# check status of VPN tunnel
show vpn ipsec sa
# commands to check VPN status/logs/information:

# monitor vpn ipsec
# show vpn debug
# show log vpn ipsec

```

#### Update VyOS config

* Manual update the key **Your_Remote_Key** or remote IP, e.g. 192.168.127.12

```
interfaces {
    dummy dum0 {
        address 127.17.12.172/32
        address 172.17.130.96/32
        description remote-vpn-ip
    }
    ethernet eth0 {
        address dhcp
        description aws-internal
        duplex auto
        hw-id 06:73:3f:28:dd:68
        smp_affinity auto
        speed auto
    }
    loopback lo {
    }
}
nat {
    source {
        rule 100 {
            description "Internal to REMOTE"
            destination {
                address 172.16.58.3/24
            }
            outbound-interface any
            source {
                address 10.104.0.0/16
            }
            translation {
                address 127.17.12.172
            }
        }
    }
}
service {
    ssh {
        disable-password-authentication
        port 22
    }
}
system {
    config-management {
        commit-revisions 20
    }
    console {
        device ttyS0 {
            speed 9600
        }
    }
    host-name vyos-vpn
    login {
        user vyos {
            authentication {
                encrypted-password "*"
                plaintext-password ""
                public-keys aws.vpn.vyos.key.io-bd:dc:ae:d6:28:b3:5f:5b:2e:43:6f:31:b8:b3:a0:58 {
                    key <KEY>
                    type ssh-rsa
                }
            }
            level admin
        }
    }
    ntp {
        server 0.pool.ntp.org {
        }
        server 1.pool.ntp.org {
        }
        server 2.pool.ntp.org {
        }
    }
    package {
        auto-sync 1
        repository community {
            components main
            distribution helium
            password ""
            url http://packages.vyos.net/vyos
            username ""
        }
    }
    syslog {
        global {
            facility all {
                level notice
            }
            facility protocols {
                level debug
            }
        }
    }
    time-zone UTC
}
vpn {
    ipsec {
        esp-group vpn-nat-esp {
            compression disable
            lifetime 28800
            mode tunnel
            pfs dh-group14
            proposal 1 {
                encryption aes256
                hash sha256
            }
        }
        ike-group vpn-nat-ike {
            dead-peer-detection {
                action restart
                interval 30
                timeout 30
            }
            ikev2-reauth no
            key-exchange ikev1
            lifetime 28800
            proposal 1 {
                dh-group 14
                encryption aes256
                hash sha256
            }
        }
        ipsec-interfaces {
            interface eth0
        }
        logging {
            log-modes all
        }
        nat-traversal enable
        site-to-site {
            peer 192.168.127.12 {
                authentication {
                    id 13.14.15.16
                    mode pre-shared-secret
                    pre-shared-secret Your_Remote_Key
                }
                connection-type initiate
                default-esp-group vpn-nat-esp
                ike-group vpn-nat-ike
                ikev2-reauth inherit
                local-address 10.104.16.128
                tunnel 0 {
                    allow-nat-networks disable
                    allow-public-networks disable
                    local {
                        prefix 127.17.12.172/32
                    }
                    remote {
                        prefix 172.16.58.3/24
                    }
                }
            }
        }
    }
}
```

* Reboot the VyOS

```
_vyatta_op_run reboot
```<file_sep>+++

title = "Java Note - 4: Date Time "
description="Java 8 provides a comprehensive Date-Time API to work with date, time, and datetime"
+++

## Date-Time API

Through the java.time packages, Java 8 provides a comprehensive Date-Time API to work with date, time, and datetime. By default, most of the classes are based on the ISO-8601 standards. The main classes are

* Instant
   * represents an instant on the timeline and it is suitable for machines, for example, as timestamps for event
* LocalDate,  LocalTime, LocalDateTime
    * represents human readable date, time, and datetime without a time zone. 

* OffsetTime, OffsetDateTime
    * It represent a time and datetime with a zone offset from UTC. 
    
* ZonedDateTime
    * It represents a datetime for a time zone with zone rules, which will adjust the time according to the daylight saving time changes in the time zone.
    
### ISO-8601 Standards for Datetime
* [date]T[time][zone offset] 
* A date component consists of three calendar fields: year, month, and day. Two fields in a date are separated by a hyphen: year-month-day\
* Epoch is Midnight January 1, 1970 UTC

### Useful Datetime-Related Enums
* Month
* DayOfWeek
* ChronoField
* ChronoUnit

## Period 

A period is an amount of time defined in terms of calendar fields years, months, and days. A duration is also an amount of time measured in terms of seconds and nanoseconds. Negative periods are supported. What is the difference between a period and a duration? A duration represents an exact number of nanoseconds, whereas a period represents an inexact amount of time. A period is for humans what a duration is for machines.

## Partial 
Partials
A partial is a date, time, or datetime that does not fully specify an instant on a timeline, but still makes sense to humans. With some more information, a partial may match multiple instants on the timeline.

## Adjusting Dates

Sometimes you want to adjust a date and time to have a particular characteristic, for example, the first Monday of the month, the next Tuesday, etc. You can perform adjustments to a date and time using an instance of the `TemporalAdjuster` interface. The interface has one method, adjustInto(), that takes a Temporal and returns a `Temporal`.

## Formatting

The most important point to keep in mind is that formatting and parsing are
always performed by an object of the DateTimeFormatter class. 

### DateTimeApiDemo

```java
import java.time.Duration;
import java.time.DayOfWeek;

import java.time.format.DateTimeFormatter;
import java.time.format.DateTimeParseException;

import java.time.Instant;
import java.time.LocalDate;
import java.time.LocalTime; 
import java.time.LocalDateTime; 

import java.time.Month;
import java.time.MonthDay;

import java.time.OffsetDateTime;
import java.time.Period;


import java.time.temporal.Temporal;
import java.time.temporal.TemporalAdjusters;
import java.time.temporal.ChronoField;
import java.time.temporal.TemporalAccessor;


import java.time.Year;
import java.time.YearMonth;

import java.time.ZoneOffset;
import java.time.ZonedDateTime;
import java.time.ZoneId;

import java.util.Locale;
import java.util.Set;


import static java.time.Month.JANUARY;
import static java.time.temporal.ChronoUnit.DAYS;
import static java.time.temporal.ChronoUnit.HOURS;
import static java.time.temporal.ChronoUnit.MINUTES;


public class DateTimeApiDemo {

    
    public static String format(Temporal co, String pattern) {
        DateTimeFormatter fmt = DateTimeFormatter.ofPattern(pattern, Locale.US);
        return fmt.format(co);
    }

    public static void parseStr(DateTimeFormatter formatter, String text) {
        try {
            TemporalAccessor ta = formatter.parseBest(text, OffsetDateTime::from, LocalDateTime::from, LocalDate::from);
            if (ta instanceof OffsetDateTime) {
                OffsetDateTime odt = OffsetDateTime.from(ta);
                System.out.println("OffsetDateTime: " + odt);
            } else if (ta instanceof LocalDateTime) {
                LocalDateTime ldt = LocalDateTime.from(ta);
                System.out.println("LocalDateTime: " + ldt);
            } else if (ta instanceof LocalDate) {
                LocalDate ld = LocalDate.from(ta);
                System.out.println("LocalDate: " + ld);
            } else {
                System.out.println("Parsing returned: " + ta);
            }
        } catch (DateTimeParseException e) {
            System.out.println(e.getMessage());
        }
    }

    public static void main(String[] args) {

        // Get current date, time, and datetime 
        LocalDate dateOnly = LocalDate.now();  // 2016-03-12
        LocalTime timeOnly = LocalTime.now();  // 09:17:56.200
        LocalDateTime dateTime = LocalDateTime.now(); // 2016-03-12T09:17:56.200
        ZonedDateTime dateTimeWithZone = ZonedDateTime.now(); // 2016-03-12T09:17:56.202+11:00[Australia/Sydney]

        //  ofXXX() method
        LocalDate ld1 = LocalDate.of(2012, 5, 2); // 2012-05-02
        LocalDate ld2 = LocalDate.of(2012, Month.JULY, 4); // 2012-07-04
        LocalDate ld3 = LocalDate.ofEpochDay(2002); // 1975-06-26
        LocalDate ld4 = LocalDate.ofYearDay(2014, 40); // 2014-02-09

        // The plusXXX( ) and minusXXX( ) Methods
        LocalDate ld = LocalDate.of(2015, 5, 2); // 2015-05-02
        LocalDate ldp1 = ld.plusDays(5); // 2015-05-07
        LocalDate ldp2 = ld.plusMonths(3); // 2015-08-02
        LocalDate ldp3 = ld.plusWeeks(3); // 2015-05-23
        LocalDate ldm1 = ld.minusMonths(7); // 2014-10-02
        LocalDate ldm2 = ld.minusWeeks(3); // 2015-04-11

        // Instant 
        Instant i1 = Instant.ofEpochSecond(20); // i1:1970-01-01T00:00:20Z
        Instant i2 = Instant.ofEpochSecond(55); // i2:1970-01-01T00:00:55Z

        Duration d1 = Duration.ofSeconds(55);
        Duration d2 = Duration.ofSeconds(-17);

        // Compare instants
        System.out.println("i1.isBefore(i2):" + i1.isBefore(i2)); // i1.isBefore(i2):true
        System.out.println("i1.isAfter(i2):" + i1.isAfter(i2)); // i1.isAfter(i2):false

        // Add and subtract durations to instants
        Instant i3 = i1.plus(d1);
        Instant i4 = i2.minus(d2);
        System.out.println("i1.plus(d1):" + i3); // i1.plus(d1):1970-01-01T00:01:15Z

        System.out.println("i2.minus(d2):" + i4);  // i2.minus(d2):1970-01-01T00:01:12Z

        // Add two durations
        System.out.println("d1.plus(d2):" + d1.plus(d2)); // d1.plus(d2):PT38S


        // Print All Zone Id
        Set<String> zoneIds = ZoneId.getAvailableZoneIds();
        for (String zoneId: zoneIds) {
          System.out.println(zoneId);
        }

        // DayOfWeek
        DayOfWeek dw1 = DayOfWeek.from(ld); // THURSDAY 

        // Chrono 

        LocalDateTime now = LocalDateTime.now();
        System.out.println("Year: " + now.get(ChronoField.YEAR));
        System.out.println("Month: " + now.get(ChronoField.MONTH_OF_YEAR));
        System.out.println("Day: " + now.get(ChronoField.DAY_OF_MONTH));
        System.out.println("Hour-of-day: " + now.get(ChronoField.HOUR_OF_DAY));
        System.out.println("Hour-of-AMPM: " + now.get(ChronoField.HOUR_OF_AMPM));
        System.out.println("AMPM-of-day: " + now.get(ChronoField.AMPM_OF_DAY));

        Period p1 = Period.of(2, 3, 5); // 2 years, 3 months, and 5 days
        Period p2 = Period.ofDays(25); // 25 days
        Period p3 = Period.ofMonths(-3); // -3 months
        Period p4 = Period.ofWeeks(3); // 3 weeks (21 days)

        // Date Adjuster
        LocalDate ld3 = ld1.with(TemporalAdjusters.next(DayOfWeek.MONDAY));
        System.out.println(ld3);
        ld3 = ld1.with(TemporalAdjusters.nextOrSame(DayOfWeek.TUESDAY));
        System.out.println(ld3);

        // Date Time Format         
        System.out.println(format(ld, "M/d/yyyy"));
        System.out.println(format(ld, "MM/dd/yyyy"));
        System.out.println(format(ld, "MMM dd, yyyy"));
        System.out.println(format(ld, "MMMM dd, yyyy"));
        System.out.println(format(ld, "EEEE, MMMM dd, yyyy"));
        System.out.println(format(ld, "'Month' q 'in' QQQ"));
        System.out.println(format(ld, "[MM-dd-yyyy][' at' HH:mm:ss]"));

        // Parse date time
        DateTimeFormatter parser = DateTimeFormatter.ofPattern("yyyy-MM-dd['T'HH:mm:ss[Z]]");
        parseStr(parser, "2012-05-31"); //  LocalDate: 2012-05-31
        parseStr(parser, "2012-05-31T16:30:12"); // LocalDateTime: 2012-05-31T16:30:12
        parseStr(parser, "2012-05-31T16:30:12-0500"); // OffsetDateTime: 2012-05-31T16:30:12-05:00
        parseStr(parser, "2012-05-31Hello"); // Text '2012-05-31Hello' could not be parsed, unparsed text found at index 10



    }
}

```
<file_sep>+++



title = "VirtualBox Notes"
description = "VirtualBox practices"
draft = false
+++


## Network


### Not attached. 

In this mode, Oracle VM VirtualBox reports to the guest that a network card is present, but that there is no connection. This is as if no Ethernet cable was plugged into the card. Using this mode, it is possible to "pull" the virtual Ethernet cable and disrupt the connection, which can be useful to inform a guest operating system that no network connection is available and enforce a reconfiguration.

### Network Address Translation (NAT). 

If all you want is to browse the Web, download files, and view email inside the guest, then this default mode should be sufficient for you, and you can skip the rest of this section. Please note that there are certain limitations when using Windows file sharing. See Section 6.3.3, “NAT Limitations”.

### NAT Network. 

A NAT network is a type of internal network that allows outbound connections. See Section 6.4, “Network Address Translation Service”.

### Bridged networking. 

This is for more advanced networking needs, such as network simulations and running servers in a guest. When enabled, Oracle VM VirtualBox connects to one of your installed network cards and exchanges network packets directly, circumventing your host operating system's network stack.

### Internal networking. 

This can be used to create a different kind of software-based network which is visible to selected virtual machines, but not to applications running on the host or to the outside world.

### Host-only networking. 

This can be used to create a network containing the host and a set of virtual machines, without the need for the host's physical network interface. Instead, a virtual network interface, similar to a loopback interface, is created on the host, providing connectivity among virtual machines and the host.


### Overview of networking modes



Mode|VM -> Host|VM <- Host|VM1<->VM2| VM->Net/LAN|VM<-Net/LAN
----|:---:|:---:|:----:|:---:|:-----:|
Host-only | + | + | + | – |  – 
Internal | – | – | + | – | – 
Bridged | + | + | + | + | +  
NAT | + | Port forward | – | + | Port forward 
NATservice | + | Port forward | + | + | Port forward

### How to SSH between HOST and VMS

- Enable `password` for SSH
- Create a new host-only network 
  - File > Network Manager > Host-only networks
  - Create a new network `vboxnet0` 
- Setup network adapter
  - Adapter 1: Bridged Network 
    - Name: wifi
  - Adapter 2: Host-only network
    - Name: vboxnet0


<file_sep>+++
title = "Data Types & Ownership"
description="Rustlang Introduction: Mutability, Shadowing, Data Types, Ownership, Borrowing "
weight = 1
+++

### Mutability

Rust encourages you to favor immutability. It’s important that we get compile-time errors when we attempt to change a value that we previously designated as immutable because this very situation can lead to bugs.

But mutability can be very useful. To make them mutable is simply adding mut in front of the variable name. In addition to allowing this value to change, mut conveys intent to future readers of the code by indicating that other parts of the code will be changing this variable value.


#### Shadowing 

* Rustaceans say that the first variable is shadowed by the second, which means that the second variable’s value is what appears when the variable is used. 

* Sample 

    ```rs
    fn main() {
        let x = 5;

        let x = x + 1;

        let x = x * 2;

        println!("The value of x is: {}", x); // 12
    }
    ```

* Shadowing is different from marking a variable as mut, because we’ll get a compile-time error if we accidentally try to reassign to this variable without using the let keyword. By using let, we can perform a few transformations on a value but have the variable be immutable after those transformations have been completed. 

* The other difference between mut and shadowing is that because we’re effectively creating a new variable when we use the let keyword again, we can change the type of the value but reuse the same name. 

### Data types

Every value in Rust is of a certain data type, which tells Rust what kind of data is being specified so it knows how to work with that data. We’ll look at two data type subsets: scalar and compound.

* A scalar type represents a single value. Rust has four primary scalar types: integers, floating-point numbers, Booleans, and characters. 

* Compound types can group multiple values into one type. Rust has two primitive compound types: tuples and arrays.

#### Type parse 

```rs
let guess: u32 = "42".parse().expect("Not a number!");
```

#### The Tuple Type

* A tuple is a general way of grouping together some number of other values with a variety of types into one compound type. 

```rs
let tup: (i32, f64, u8) = (500, 6.4, 1);
let tup = (500, 6.4, 1);
let (x, y, z) = tup;
println!("The value of y is: {}", y); //6.4
```


### Ownership

Rust’s central feature is ownership. Although the feature is straightforward to explain, it has deep implications for the rest of the language.


#### Ownership Rules

* Each value in Rust has a variable that’s called its owner.
* There can only be one owner at a time.
* When the owner goes out of scope, the value will be dropped.


#### Variable Scope

* A scope is the range within a program for which an item is valid. 

#### Memory and Allocation

* Rust takes a different path: the memory is automatically returned once the variable that owns it goes out of scope. 

* Explanation of memory allocation and free

```go
{
    let s = String::from("hello"); // s is valid from this point forward
}   // this scope is now over, and s is no longer valid
```

* There is a natural point at which we can return the memory our String needs to the operating system: when s goes out of scope. When a variable goes out of scope, Rust calls a special function for us. This function is called drop, and it’s where the author of String can put the code to return the memory. Rust calls drop automatically at the closing curly bracket.

#### Move or Clone or Copy 

* Sample of move; the scalar type has no this problem

```rs
let s1 = String::from("hello");
let s2 = s1;  // move value from s1 to s2
              // s1 is no longer valid 

println!("{}, world!", s1); // Compile error - value used here after move
```
* If we do want to deeply copy the heap data of the String, not just the stack data, we can use a common method called clone. 

```rs
let s2 = s1.clone();
```

* Rust has a special annotation called the Copy trait that we can place on types like integers that are stored on the stack. If a type has the Copy trait, an older variable is still usable after assignment. Rust won’t let us annotate a type with the Copy trait if the type, or any of its parts, has implemented the Drop trait. If the type needs something special to happen when the value goes out of scope and we add the Copy annotation to that type, we’ll get a compile-time error. 

* Types for copy
    
    * All the integer types, such as u32.
    * The Boolean type, bool, with values true and false.
    * All the floating point types, such as f64.
    * The character type, char.
    * Tuples, if they only contain types that are also Copy. For example, (i32, i32) is Copy, but (i32, String) is not.


### References & Borrowing

* At any given time, you can have either one mutable reference or any number of immutable references.
* References must always be valid.

#### Slice

* Another data type that does not have ownership is the slice. Slices let you reference a contiguous sequence of elements in a collection rather than the whole collection.


### Struct

* Structs are similar to tuples, which were discussed in Chapter 3. Like tuples, the pieces of a struct can be different types. Unlike with tuples, you’ll name each piece of data so it’s clear what the values mean. 

* sample code of strut

    ```rs 
    struct User {
        username: String,
        email: String,
        sign_in_count: u64,
        active: bool,
    }


    fn build_user(email: String, username: String) -> User {
        User {
            email: email,
            username: username,
            active: true,
            sign_in_count: 1,
        }
    }
    ```

* Creating Instances From Other Instances 

    ```rs
    let user2 = User {
        email: String::from("<EMAIL>"),
        username: String::from("anotherusername567"),
        ..user1
    };
    ```

#### Unit-Like Struct

* structs that don’t have any fields! These are called unit-like structs because they behave similarly to (), the unit type. 

#### Methods

* Methods are similar to functions: they’re declared with the fn keyword and their name, they can have parameters and a return value, and they contain some code that is run when they’re called from somewhere else. However, methods are different from functions in that they’re defined within the context of a struct (or an enum or a trait object. Their first parameter is always self, which represents the instance of the struct the method is being called on.

    ```rs
    #[derive(Debug)]
    struct Rectangle {
        width: u32,
        height: u32,
    }

    impl Rectangle {
        fn area(&self) -> u32 {
            self.width * self.height
        }

        fn can_hold(&self, other: &Rectangle) -> bool {
            self.width > other.width && self.height > other.height
        }
    }

    fn main() {
        let rect1 = Rectangle {
            width: 30,
            height: 50,
        };

        println!(
            "The area of the rectangle is {} square pixels.",
            rect1.area()
        );

        let rect2 = Rectangle {
            width: 10,
            height: 40,
        };
        let rect3 = Rectangle {
            width: 60,
            height: 45,
        };

        println!("Can rect1 hold rect2? {}", rect1.can_hold(&rect2));
        println!("Can rect1 hold rect3? {}", rect1.can_hold(&rect3));
    }
    ```

### Enum & Option

* Enums allow you to define a type by enumerating its possible values. First, we’ll define and use an enum to show how an enum can encode meaning along with data. 

* A particularly useful enum, called Option, which expresses that a value can be either something or nothing. 

* Pattern matching in the match expression makes it easy to run different code for different values of an enum.

* Sample from Rust standard library

    ```rs
    struct Ipv4Addr {
        // --snip--
    }

    struct Ipv6Addr {
        // --snip--
    }

    enum IpAddr {
        V4(Ipv4Addr),
        V6(Ipv6Addr),
    }
    ```

* Rust does not have nulls, but it does have an enum that can encode the concept of a value being present or absent. This enum is Option<T>

    ```rs
    enum Option<T> {
        Some(T),
        None,
    }
    ```

#### Match

* Rust has an extremely powerful control flow operator called match that allows you to compare a value against a series of patterns and then execute code based on which pattern matches. Patterns can be made up of literal values, variable names, wildcards, and many other things; 

    ```rs

    #![allow(unused_variables)]
    fn main() {
        enum Coin {
            Penny,
            Nickel,
            Dime,
            Quarter,
        }

        fn value_in_cents(coin: Coin) -> u8 {
            match coin {
                Coin::Penny => 1,
                Coin::Nickel => 5,
                Coin::Dime => 10,
                Coin::Quarter => 25,
            }
        }
    }
    ```

* Match with Option<T>

    ```rs
    #![allow(unused_variables)]
    fn main() {
    fn plus_one(x: Option<i32>) -> Option<i32> {
        match x {
            None => None,
            Some(i) => Some(i + 1),
        }
    }

    let five = Some(5);
    let six = plus_one(five);
    let none = plus_one(None);
    }
    ```


#### Matches Are Exhaustive

* Rust knows that we didn’t cover every possible case and even knows which pattern we forgot! Matches in Rust are exhaustive: we must exhaust every last possibility in order for the code to be valid. Especially in the case of Option<T>, when Rust prevents us from forgetting to explicitly handle the None case.


* The _ Placeholder

    ```rs
    let some_u8_value = 0u8;
    match some_u8_value {
        1 => println!("one"),
        3 => println!("three"),
        5 => println!("five"),
        7 => println!("seven"),
        _ => (),
    }
    ```

### Control Flow with if let

* The if let syntax lets you combine if and let into a less verbose way to handle values that match one pattern while ignoring the rest. 

    ```rs
    #![allow(unused_variables)]
    fn main() {
        let some_u8_value = Some(0u8);
        match some_u8_value {
            Some(3) => println!("three"),
            _ => (),
        }
    }
    ```
* with else

    ```rs
    let mut count = 0;
    if let Coin::Quarter(state) = coin {
        println!("State quarter from {:?}!", state);
    } else {
        count += 1;
    }
    ```







<file_sep>+++
title = "Adv Bash - 2"
description = "Reference Cards - String Manipulation"
+++

### Manipulating Strings

Bash supports a surprising number of string manipulation operations. Unfortunately, these tools lack a unified focus. Some are a subset of parameter substitution, and others fall under the functionality of the UNIX expr command. This results in inconsistent command syntax and overlap of functionality, not to mention confusion.


####  String Operations

Expression	| Meaning
|-----|-----
| ${#string}|	Length of $string
| ${string:position}|	Extract substring from $string at $position
| ${string:position:length}|	Extract $length characters substring from $string at $position [zero-indexed, | first character is at position 0]
| ${string#substring}|	Strip shortest match of $substring from front of $string
| ${string##substring}|	Strip longest match of $substring from front of $string
| ${string%substring}|	Strip shortest match of $substring from back of $string
| ${string%%substring}|	Strip longest match of $substring from back of $string
| ${string/substring/replacement}|	Replace first match of $substring with $replacement
| ${string//substring/replacement}|	Replace all matches of $substring with $replacement
| ${string/#substring/replacement}|	If $substring matches front end of $string, substitute $replacement for  $substring
| ${string/%substring/replacement}|	If $substring matches back end of $string, substitute $replacement for $substring
|expr match "$string" '$substring'	| Length of matching $substring* at beginning of $string
|expr "$string" : '$substring'	| Length of matching $substring* at beginning of $string
|expr index "$string" $substring	| Numerical position in $string of first character in $substring* that matches [0 if no match, first character counts as position 1]
|expr substr $string $position $length	| Extract$length characters from $string starting at $position [0 if |no match, first character counts as position 1]
|expr match "$string" '\($substring\)'	| Extract$substring*, searching from beginning of $string
|expr "$string" : '\($substring\)'	| Extract$substring* , searching from beginning of $string
|expr match "$string" '.*\($substring\)'	| Extract$substring*, searching from end of $string
|expr "$string" : '.*\($substring\)'	| Extract$substring*, searching from end of $string





#### Parameter Substitution and Expansion

Expression |	Meaning
|------|--------
| ${var} | Value of var (same as $var)
| ${var-$DEFAULT} |	If var not set, evaluate expression as $DEFAULT *
| ${var:-$DEFAULT} |	If var not set or is empty, evaluate expression as $DEFAULT *
| ${var=$DEFAULT} |	If var not set, evaluate expression as $DEFAULT *
| ${var:=$DEFAULT} |	If var not set or is empty, evaluate expression as $DEFAULT *
| ${var+$OTHER} |	If var set, evaluate expression as $OTHER, otherwise as null string
| ${var:+$OTHER} |	If var set, evaluate expression as $OTHER, otherwise as null string
| ${var?$ERR_MSG} |	If var not set, print $ERR_MSG and abort script with an exit status of 1.*
| ${var:?$ERR_MSG} |	If var not set, print $ERR_MSG and abort script with an exit status of 1.*
| ${!varprefix*} |	Matches all previously declared variables beginning with varprefix
| ${!varprefix@} |	Matches all previously declared variables beginning with varprefix



#### Samples

* Replace file name from abc_noteN to abc_N

```bash
touch abc_note1 abc_note2 abc_note3

for f in $(find . -type f -name *abc_note*); 
do 
    mv -- $f "${f//note}" 
done ;

ls abc*
# expected output
# abc_1   abc_2   abc_3
```


















<file_sep>+++

title = "Frameworks"
weight = 7
+++

{{%children style="card" description="true" sort="Weight" %}}<file_sep>+++
title = "MySql: JSON"
description="Json Support "
+++

## Json support 

As of MySQL 5.7.8, MySQL supports a native JSON data type defined by RFC 7159 that enables efficient access to data in JSON (JavaScript Object Notation) documents. The JSON data type provides these advantages over storing JSON-format strings in a string column:

* Automatic validation of JSON documents stored in JSON columns. Invalid documents produce an error.

* Optimized storage format. JSON documents stored in JSON columns are converted to an internal format that permits quick read access to document elements. When the server later must read a JSON value stored in this binary format, the value need not be parsed from a text representation. The binary format is structured to enable the server to look up subobjects or nested values directly by key or array index without reading all values before or after them in the document.



### Caveats

* The size of any JSON document stored in a JSON column is limited to the value of the max_allowed_packet system variable. 

* A JSON column cannot have a non-NULL default value.


### Json functions

> Along with the JSON data type, a set of SQL functions is available to enable operations on JSON values, such as creation, manipulation, and searching. 


Name |  Description
:--|:--|
->  | Return value from JSON column after evaluating path; equivalent to JSON_EXTRACT().
->>  | Return value from JSON column after evaluating path and unquoting the result; equivalent to JSON_UNQUOTE(JSON_EXTRACT()).
JSON_APPEND() |  (deprecated 5.7.9)  Append data to JSON document
JSON_ARRAY() |   Create JSON array
JSON_ARRAY_APPEND() |   Append data to JSON document
JSON_ARRAY_INSERT() |   Insert into JSON array
JSON_CONTAINS() |   Whether JSON document contains specific object at path
JSON_CONTAINS_PATH() |   Whether JSON document contains any data at path
JSON_DEPTH() |   Maximum depth of JSON document
JSON_EXTRACT() |   Return data from JSON document
JSON_INSERT() |   Insert data into JSON document
JSON_KEYS() |   Array of keys from JSON document
JSON_LENGTH() |   Number of elements in JSON document
JSON_MERGE() |  (deprecated 5.7.22)  Merge JSON documents, preserving duplicate keys. Deprecated synonym for JSON_MERGE_PRESERVE() | 
JSON_MERGE_PATCH() |   Merge JSON documents, replacing values of duplicate keys
JSON_MERGE_PRESERVE() |   Merge JSON documents, preserving duplicate keys
JSON_OBJECT() |   Create JSON object
JSON_PRETTY() |   Print a JSON document in human-readable format
JSON_QUOTE() |   Quote JSON document
JSON_REMOVE() |   Remove data from JSON document
JSON_REPLACE() |   Replace values in JSON document
JSON_SEARCH() |   Path to value within JSON document
JSON_SET() |   Insert data into JSON document
JSON_STORAGE_SIZE() |   Space used for storage of binary representation of a JSON document
JSON_TYPE() |   Type of JSON value
JSON_UNQUOTE() |   Unquote JSON value
JSON_VALID() |   Whether JSON value is valid


### Export table to json file

* Return json object

```sql
---  Return record as json in query
SELECT 
json_object( 
   'entityId',  entityId,
   'categoryName',  categoryName,
   'description',  description,
   'picture',  picture
 ) as json, 'Category' 
  FROM  Category 
  WHERE entityId = 1 ; 
```

* Json object result

```json
{
  "picture": null,
  "entityId": 1,
  "description": "Soft drinks, coffees, teas, beers, and ales",
  "categoryName": "Beverages"
}

```

* Return record as json array


```sql
-- Use json_array and json_object functions 
DROP TEMPORARY TABLE IF EXISTS tmp_json_data;

CREATE TEMPORARY TABLE tmp_json_data (
  jsonText TEXT
) 
SELECT 
json_object( 
   'entityId',  entityId,
   'categoryName',  categoryName,
   'description',  description,
   'picture',  picture
 ) as json 
  INTO jsonText
FROM  Category ; 
  
SET SESSION  group_concat_max_len = 9999;

SELECT concat('[', group_concat(jsonText),']') jsonArray
FROM tmp_json_data;
```

* Json array result

```json
[
    {
      "categoryName" : "Beverages",
      "description" : "Soft drinks, coffees, teas, beers, and ales",
      "entityId" : 1,
      "picture" : null
    },
    {
      "categoryName" : "Condiments",
      "description" : "Sweet and savory sauces, relishes, spreads, and seasonings",
      "entityId" : 2,
      "picture" : null
    },
    {
      "categoryName" : "Confections",
      "description" : "Desserts, candies, and sweet breads",
      "entityId" : 3,
      "picture" : null
    }
]

```
<file_sep>+++
title = "F# Pattern Matching"
description = "F# Pattern Matching"
weight = 8
+++


## Pattern

Patterns are rules for transforming input data. They are used throughout F# to compare data with a logical structure or structures, decompose data into constituent parts, or extract information from data in various ways.



Name | Description | Example
--- | --- | ---
Constant pattern | Any numeric, character, or string literal, an enumeration constant, or a defined literal identifier | 1.0, "test", 30, Color.Red
Identifier pattern | A case value of a discriminated union, an exception label, or an active pattern case | Some(x) Failure(msg)
Variable pattern | identifier | a
as pattern | pattern as identifier | (a, b) as tuple1
OR pattern | pattern1 | pattern2 | ([h] | [h; _])
AND pattern | pattern1 & pattern2 | (a, b) & (_, "test")
Cons pattern | identifier :: list-identifier | h :: t
List pattern | [ pattern_1; ... ; pattern_n ] | [ a; b; c ]
Array pattern | [| pattern_1; ..; pattern_n |] | [| a; b; c |]
Parenthesized pattern | ( pattern ) | ( a )
Tuple pattern | ( pattern_1, ... , pattern_n ) | ( a, b )
Record pattern | { identifier1 = pattern_1; ... ; identifier_n = pattern_n } | { Name = name; }
Wildcard pattern | _ | _
Pattern together with type annotation | pattern : type | a : int
Type test pattern | :? type [ as identifier ] | :? System.DateTime as dt
Null pattern | null | null
Nameof pattern | nameof expr | nameof str


### Constant Patterns

```fsharp

[<Literal>]
let Three = 3

let filter123 x =
    match x with
    // The following line contains literal patterns combined with an OR pattern.
    | 1 | 2 | Three -> printfn "Found 1, 2, or 3!"
    // The following line contains a variable pattern.
    | var1 -> printfn "%d" var1

for x in 1..10 do filter123 x
// Found 1, 2, or 3!
// Found 1, 2, or 3!
// Found 1, 2, or 3!
// 4
// 5
// 6
// 7
// 8
// 9
// 10


type Color =
    | Red = 0
    | Green = 1
    | Blue = 2

let printColorName (color:Color) =
    match color with
    | Color.Red -> printfn "Red"
    | Color.Green -> printfn "Green"
    | Color.Blue -> printfn "Blue"
    | _ -> ()

printColorName Color.Red
printColorName Color.Green
printColorName Color.Blue
// Red
// Green
// Blue

```

### Identifier Patterns

```fsharp
let printOption (data : int option) =
    match data with
    | Some var1  -> printfn "%d" var1
    | None -> ()

type PersonName =
    | FirstOnly of string
    | LastOnly of string
    | FirstLast of string * string

let constructQuery personName =
    match personName with
    | FirstOnly(firstName) -> printfn "May I call you %s?" firstName
    | LastOnly(lastName) -> printfn "Are you Mr. or Ms. %s?" lastName
    | FirstLast(firstName, lastName) -> printfn "Are you %s %s?" firstName lastName

constructQuery (FirstOnly("john"))
constructQuery (LastOnly("smith"))
constructQuery (FirstLast("john","smith"))
// May I call you john?
// Are you Mr. or Ms. smith?
// Are you <NAME>?

```


### Variable Patterns

```fsharp
let function1 x =
    match x with
    | (var1, var2) when var1 > var2 -> printfn "%d is greater than %d" var1 var2
    | (var1, var2) when var1 < var2 -> printfn "%d is less than %d" var1 var2
    | (var1, var2) -> printfn "%d equals %d" var1 var2

function1 (1,2)
function1 (2, 1)
function1 (0, 0)
// 1 is less than 2
// 2 is greater than 1
// 0 equals 0
```

### as Pattern

The `as` pattern is a pattern that has an as clause appended to it.

```fsharp
let (var1, var2) as tuple1 = (1, 2)
printfn "%d %d %A" var1 var2 tuple1
// 1 2 (1, 2)
```


### OR Pattern

The OR pattern is used when input data can match multiple patterns, and you want to execute the same code as a result. 

```fsharp
let detectZeroOR point =
    match point with
    | (0, 0) | (0, _) | (_, 0) -> printfn "Zero found."
    | _ -> printfn "Both nonzero."
detectZeroOR (0, 0)
detectZeroOR (1, 0)
detectZeroOR (0, 10)
detectZeroOR (10, 15)
// output
// Zero found.
// Zero found.
// Zero found.
// Both nonzero.
```



### AND Pattern

The AND pattern requires that the input match two patterns. The types of both sides of the AND pattern must be compatible.

```fsharp
let detectZeroAND point =
    match point with
    | (0, 0) -> printfn "Both values zero."
    | (var1, var2) & (0, _) -> printfn "First value is 0 in (%d, %d)" var1 var2
    | (var1, var2)  & (_, 0) -> printfn "Second value is 0 in (%d, %d)" var1 var2
    | _ -> printfn "Both nonzero."
detectZeroAND (0, 0)
detectZeroAND (1, 0)
detectZeroAND (0, 10)
detectZeroAND (10, 15)
// output
// Both values zero.
// Second value is 0 in (1, 0)
// First value is 0 in (0, 10)
// Both nonzero.
```



### Cons Pattern

The cons pattern is used to decompose a list into the first element, the head, and a list that contains the remaining elements, the tail.

```fsharp
let list1 = [ 1; 2; 3; 4 ]

// This example uses a cons pattern and a list pattern.
let rec printList l =
    match l with
    | head :: tail -> printf "%d " head; printList tail
    | [] -> printfn ""

printList list1
```

### List Pattern

The list pattern enables lists to be decomposed into a number of elements. The list pattern itself can match only lists of a specific number of elements.


```fsharp
// This example uses a list pattern.
let listLength list =
    match list with
    | [] -> 0
    | [ _ ] -> 1
    | [ _; _ ] -> 2
    | [ _; _; _ ] -> 3
    | _ -> List.length list

printfn "%d" (listLength [ 1 ])
printfn "%d" (listLength [ 1; 1 ])
printfn "%d" (listLength [ 1; 1; 1; ])
printfn "%d" (listLength [ ] )
// output
// 1
// 2
// 3
// 0
```


### Array Pattern

The array pattern resembles the list pattern and can be used to decompose arrays of a specific length.


```fsharp
// This example uses array patterns.
let vectorLength vec =
    match vec with
    | [| var1 |] -> var1
    | [| var1; var2 |] -> sqrt (var1*var1 + var2*var2)
    | [| var1; var2; var3 |] -> sqrt (var1*var1 + var2*var2 + var3*var3)
    | _ -> failwith (sprintf "vectorLength called with an unsupported array size of %d." (vec.Length))

printfn "%f" (vectorLength [| 1. |])
printfn "%f" (vectorLength [| 1.; 1. |])
printfn "%f" (vectorLength [| 1.; 1.; 1.; |])
printfn "%f" (vectorLength [| |] )
// output
// 1.000000
// 1.414214
// 1.732051
// System.Exception: vectorLength called with an unsupported array size of 0.
//    at FSI_0055.vectorLength(Double[] vec)
//    at <StartupCode$FSI_0055>.$FSI_0055.main@()
// Stopped due to error

```

### Tuple Pattern

The tuple pattern matches input in tuple form and enables the tuple to be decomposed into its constituent elements by using pattern matching variables for each position in the tuple.

```fsharp
let detectZeroTuple point =
    match point with
    | (0, 0) -> printfn "Both values zero."
    | (0, var2) -> printfn "First value is 0 in (0, %d)" var2
    | (var1, 0) -> printfn "Second value is 0 in (%d, 0)" var1
    | _ -> printfn "Both nonzero."
detectZeroTuple (0, 0)
detectZeroTuple (1, 0)
detectZeroTuple (0, 10)
detectZeroTuple (10, 15)
// output
// Both values zero.
// Second value is 0 in (1, 0)
// First value is 0 in (0, 10)
// Both nonzero.
```


### Record Pattern

The record pattern is used to decompose records to extract the values of fields. The pattern does not have to reference all fields of the record; any omitted fields just do not participate in matching and are not extracted.

```fsharp

let IsMatchByName record1 (name: string) =
    match record1 with
    | { MyRecord.Name = nameFound; MyRecord.ID = _; } when nameFound = name -> true
    | _ -> false

let recordX = { Name = "Parker"; ID = 10 }
let isNameMatched1 = IsMatchByName recordX "Parker"
let isNameMatched2 = IsMatchByName recordX "Hartono"
printfn "isNameMatched1 %A isNameMatched2 %A " isNameMatched1 isNameMatched2 
// output
// isNameMatched1 true isNameMatched2 false

let IsMatchByNameOrId record1 (name: string, id: int) =
    match record1 with
    | { MyRecord.Name = nameFound; MyRecord.ID = _; } when nameFound = name -> true
    | { MyRecord.ID = myid; MyRecord.Name = _ ;} when myid = id -> true
    | _ -> false

let isAnyMatched1 = IsMatchByNameOrId recordX ("Parker", 2)
let isAnyMatched2 = IsMatchByNameOrId recordX ("Hartono", 10)
let isAnyMatched3 = IsMatchByNameOrId recordX ("XXX", 5)
printfn "isAnyMatched1 %A isAnyMatched2 %A isAnyMatched3 %A" isAnyMatched1 isAnyMatched2 isAnyMatched3
// output
// isAnyMatched1 true isAnyMatched2 true isAnyMatched3 false

```


### Patterns That Have Type Annotations

Patterns can have type annotations. 
```fsharp
let detect1 x =
    match x with
    | 1 -> printfn "Found a 1!"
    | (var1 : int) -> printfn "%d" var1
detect1 5
detect1 1
// output
// 5
// Found a 1!
```

### Type Test Pattern

The type test pattern is used to match the input against a type. 

```fsharp
open System.Windows.Forms

let RegisterControl(control:Control) =
    match control with
    | :? Button as button -> button.Text <- "Registered."
    | :? CheckBox as checkbox -> checkbox.Text <- "Registered."
    | _ -> ()

// ------------------------------------------------------------------------
// If you're only checking if an identifier is of a particular derived type, 
// you don't need the as identifier part of the pattern
type A() = class end
type B() = inherit A()
type C() = inherit A()

let m (a: A) =
    match a with
    | :? B -> printfn "It's a B"
    | :? C -> printfn "It's a C"
    | _ -> ()

```

### Null Pattern

The null pattern matches the null value that can appear when you are working with types that allow a null value. 


```fsharp
let ReadFromFile (reader : System.IO.StreamReader) =
    match reader.ReadLine() with
    | null -> printfn "\n"; false
    | line -> printfn "%s" line; true

let fs = System.IO.File.Open("./test.fs", System.IO.FileMode.Open)
let sr = new System.IO.StreamReader(fs)
while ReadFromFile(sr) = true do ()
sr.Close()
// output
// let ReadFromFile (reader : System.IO.StreamReader) =
//     match reader.ReadLine() with
//     | null -> printfn "\n"; false
//     | line -> printfn "%s" line; true

// let fs = System.IO.File.Open("./test.fs", System.IO.FileMode.Open)
// let sr = new System.IO.StreamReader(fs)
// while ReadFromFile(sr) = true do ()
// sr.Close()
```


### Nameof pattern

The nameof pattern matches against a string when its value is equal to the expression that follows the nameof keyword.

```fsharp
let f (str: string) =
    match str with
    | nameof str -> "It's 'str'!"
    | _ -> "It is not 'str'!"

f "str" // matches
f "asdf" // does not match
// output
// It's 'str'!
// It is not 'str'!
```





<file_sep>+++
title = "Hacks"
weight = 4
+++

{{%children style="card" description="true" sort="Weight" %}}<file_sep>+++

title = "C Lecture - 2"
description = "Exercise 32 ~ 40"
+++

Author: <NAME>

All content comes from Zed's [Lecture Repository](https://github.com/zedshaw/learn-c-the-hard-way-lectures.git) and [Libraries Repository](https://github.com/zedshaw/liblcthw).  All credit goes to Zed.


### Exercise 32 Double Linked Lists

The Plan

Learn about your very first data structure:

Double Linked Lists

Creating A liblcthw Project

We'll need a project for the rest of the book called *liblcthw*.

Algorithms and Data Structures

A big step in going from amateur to professional is learning
about data structures and algorithms.

A double linked list is the easiest one.

Double Linked Lists Visually

I'll quickly draw some diagrams to show you how they work.

Automated Testing Demo

You can enter the code just fine, but watch me write
the test.

Code Reviews

.\ex32\list.h

```c
#ifndef lcthw_List_h
#define lcthw_List_h

#include <stdlib.h>

struct ListNode;

typedef struct ListNode {
    struct ListNode *next;
    struct ListNode *prev;
    void *value;
} ListNode;

typedef struct List {
    int count;
    ListNode *first;
    ListNode *last;
} List;

List *List_create();
void List_destroy(List * list);
void List_clear(List * list);
void List_clear_destroy(List * list);

#define List_count(A) ((A)->count)
#define List_first(A) ((A)->first != NULL ? (A)->first->value : NULL)
#define List_last(A) ((A)->last != NULL ? (A)->last->value : NULL)

void List_push(List * list, void *value);
void *List_pop(List * list);

void List_unshift(List * list, void *value);
void *List_shift(List * list);

void *List_remove(List * list, ListNode * node);

#define LIST_FOREACH(L, S, M, V) ListNode *_node = NULL;\
                                                   ListNode *V = NULL;\
for(V = _node = L->S; _node != NULL; V = _node = _node->M)

#endif

```

.\ex32\list.c

```c
#include <lcthw/list.h>
#include <lcthw/dbg.h>

List *List_create()
{
    return calloc(1, sizeof(List));
}

void List_destroy(List * list)
{
    LIST_FOREACH(list, first, next, cur) {
        if (cur->prev) {
            free(cur->prev);
        }
    }

    free(list->last);
    free(list);
}

void List_clear(List * list)
{
    LIST_FOREACH(list, first, next, cur) {
        free(cur->value);
    }
}

void List_clear_destroy(List * list)
{
    List_clear(list);
    List_destroy(list);
}

void List_push(List * list, void *value)
{
    ListNode *node = calloc(1, sizeof(ListNode));
    check_mem(node);

    node->value = value;

    if (list->last == NULL) {
        list->first = node;
        list->last = node;
    } else {
        list->last->next = node;
        node->prev = list->last;
        list->last = node;
    }

    list->count++;

error:
    return;
}

void *List_pop(List * list)
{
    ListNode *node = list->last;
    return node != NULL ? List_remove(list, node) : NULL;
}

void List_unshift(List * list, void *value)
{
    ListNode *node = calloc(1, sizeof(ListNode));
    check_mem(node);

    node->value = value;

    if (list->first == NULL) {
        list->first = node;
        list->last = node;
    } else {
        node->next = list->first;
        list->first->prev = node;
        list->first = node;
    }

    list->count++;

error:
    return;
}

void *List_shift(List * list)
{
    ListNode *node = list->first;
    return node != NULL ? List_remove(list, node) : NULL;
}

void *List_remove(List * list, ListNode * node)
{
    void *result = NULL;

    check(list->first && list->last, "List is empty.");
    check(node, "node can't be NULL");

    if (node == list->first && node == list->last) {
        list->first = NULL;
        list->last = NULL;
    } else if (node == list->first) {
        list->first = node->next;
        check(list->first != NULL,
                "Invalid list, somehow got a first that is NULL.");
        list->first->prev = NULL;
    } else if (node == list->last) {
        list->last = node->prev;
        check(list->last != NULL,
                "Invalid list, somehow got a next that is NULL.");
        list->last->next = NULL;
    } else {
        ListNode *after = node->next;
        ListNode *before = node->prev;
        after->prev = before;
        before->next = after;
    }

    list->count--;
    result = node->value;
    free(node);

error:
    return result;
}

```

.\ex32\list_tests.c

```c
#include "minunit.h"
#include <lcthw/list.h>
#include <assert.h>

static List *list = NULL;
char *test1 = "test1 data";
char *test2 = "test2 data";
char *test3 = "test3 data";

char *test_create()
{
    list = List_create();
    mu_assert(list != NULL, "Failed to create list.");

    return NULL;
}

char *test_destroy()
{
    List_clear_destroy(list);

    return NULL;

}

char *test_push_pop()
{
    List_push(list, test1);
    mu_assert(List_last(list) == test1, "Wrong last value.");

    List_push(list, test2);
    mu_assert(List_last(list) == test2, "Wrong last value");

    List_push(list, test3);
    mu_assert(List_last(list) == test3, "Wrong last value.");
    mu_assert(List_count(list) == 3, "Wrong count on push.");

    char *val = List_pop(list);
    mu_assert(val == test3, "Wrong value on pop.");

    val = List_pop(list);
    mu_assert(val == test2, "Wrong value on pop.");

    val = List_pop(list);
    mu_assert(val == test1, "Wrong value on pop.");
    mu_assert(List_count(list) == 0, "Wrong count after pop.");

    return NULL;
}

char *test_unshift()
{
    List_unshift(list, test1);
    mu_assert(List_first(list) == test1, "Wrong first value.");

    List_unshift(list, test2);
    mu_assert(List_first(list) == test2, "Wrong first value");

    List_unshift(list, test3);
    mu_assert(List_first(list) == test3, "Wrong last value.");
    mu_assert(List_count(list) == 3, "Wrong count on unshift.");

    return NULL;
}

char *test_remove()
{
    // we only need to test the middle remove case since push/shift 
    // already tests the other cases

    char *val = List_remove(list, list->first->next);
    mu_assert(val == test2, "Wrong removed element.");
    mu_assert(List_count(list) == 2, "Wrong count after remove.");
    mu_assert(List_first(list) == test3, "Wrong first after remove.");
    mu_assert(List_last(list) == test1, "Wrong last after remove.");

    return NULL;
}

char *test_shift()
{
    mu_assert(List_count(list) != 0, "Wrong count before shift.");

    char *val = List_shift(list);
    mu_assert(val == test3, "Wrong value on shift.");

    val = List_shift(list);
    mu_assert(val == test1, "Wrong value on shift.");
    mu_assert(List_count(list) == 0, "Wrong count after shift.");

    return NULL;
}

char *all_tests()
{
    mu_suite_start();

    mu_run_test(test_create);
    mu_run_test(test_push_pop);
    mu_run_test(test_unshift);
    mu_run_test(test_remove);
    mu_run_test(test_shift);
    mu_run_test(test_destroy);

    return NULL;
}

RUN_TESTS(all_tests);

```

Later videos will demonstrate how to code review to make code solid.

Improving It

* You can make ``List_clear_destroy`` more efficient by using
  ``LIST_FOREACH`` and doing both ``free`` calls inside one
  loop.
* You can add asserts for preconditions so that the program isn't given a ``NULL``
  value for the ``List *list`` parameters.

Improving It

* You can add invariants that check that the list's contents are always correct,
  such as ``count`` is never ``< 0``, and if ``count > 0``, then ``first`` isn't NULL.
* You can add documentation to the header file in the form of comments before
  each struct, function, and macro that describes what it does.

Extra Credit

* Research doubly vs. singly linked lists and when one is preferred over
  the other.
* Research the limitations of a doubly linked list.  For example, while they
  are efficient for inserting and deleting elements, they are very slow for
  iterating over them all.
* What operations are missing that you can imagine needing?  Some examples
  are copying, joining, and splitting.  Implement these operations and write the
  unit tests for them.

### Exercise 33 Linked List Algorithms

The Plan

Learn two sorting algorithms for double linked lists.

Watch how to conduct a simple code review.

The Code

.\ex33\list_algos.h

```c
#ifndef lcthw_List_algos_h
#define lcthw_List_algos_h

#include <lcthw/list.h>

typedef int (*List_compare) (const void *a, const void *b);

int List_bubble_sort(List * list, List_compare cmp);

List *List_merge_sort(List * list, List_compare cmp);

#endif

```

.\ex33\list_algos.c

```c
#include <lcthw/list_algos.h>
#include <lcthw/dbg.h>

inline void ListNode_swap(ListNode * a, ListNode * b)
{
    void *temp = a->value;
    a->value = b->value;
    b->value = temp;
}

int List_bubble_sort(List * list, List_compare cmp)
{
    int sorted = 1;

    if (List_count(list) <= 1) {
        return 0;        // already sorted
    }

    do {
        sorted = 1;
        LIST_FOREACH(list, first, next, cur) {
            if (cur->next) {
                if (cmp(cur->value, cur->next->value) > 0) {
                    ListNode_swap(cur, cur->next);
                    sorted = 0;
                }
            }
        }
    } while (!sorted);

    return 0;
}

inline List *List_merge(List * left, List * right, List_compare cmp)
{
    List *result = List_create();
    void *val = NULL;

    while (List_count(left) > 0 || List_count(right) > 0) {
        if (List_count(left) > 0 && List_count(right) > 0) {
            if (cmp(List_first(left), List_first(right)) <= 0) {
                val = List_shift(left);
            } else {
                val = List_shift(right);
            }

            List_push(result, val);
        } else if (List_count(left) > 0) {
            val = List_shift(left);
            List_push(result, val);
        } else if (List_count(right) > 0) {
            val = List_shift(right);
            List_push(result, val);
        }
    }

    return result;
}

List *List_merge_sort(List * list, List_compare cmp)
{
    List *result = NULL;

    if (List_count(list) <= 1) {
        return list;
    }

    List *left = List_create();
    List *right = List_create();
    int middle = List_count(list) / 2;

    LIST_FOREACH(list, first, next, cur) {
        if (middle > 0) {
            List_push(left, cur->value);
        } else {
            List_push(right, cur->value);
        }

        middle--;
    }

    List *sort_left = List_merge_sort(left, cmp);
    List *sort_right = List_merge_sort(right, cmp);

    if (sort_left != left)
        List_destroy(left);
    if (sort_right != right)
        List_destroy(right);

    result = List_merge(sort_left, sort_right, cmp);

    List_destroy(sort_left);
    List_destroy(sort_right);

    return result;
}

```

.\ex33\list_algos_tests.c

```c
#include "minunit.h"
#include <lcthw/list_algos.h>
#include <assert.h>
#include <string.h>

char *values[] = { "XXXX", "1234", "abcd", "xjvef", "NDSS" };

#define NUM_VALUES 5

List *create_words()
{
    int i = 0;
    List *words = List_create();

    for (i = 0; i < NUM_VALUES; i++) {
        List_push(words, values[i]);
    }

    return words;
}

int is_sorted(List * words)
{
    LIST_FOREACH(words, first, next, cur) {
        if (cur->next && strcmp(cur->value, cur->next->value) > 0) {
            debug("%s %s", (char *)cur->value,
                    (char *)cur->next->value);
            return 0;
        }
    }

    return 1;
}

char *test_bubble_sort()
{
    List *words = create_words();

    // should work on a list that needs sorting
    int rc = List_bubble_sort(words, (List_compare) strcmp);
    mu_assert(rc == 0, "Bubble sort failed.");
    mu_assert(is_sorted(words),
            "Words are not sorted after bubble sort.");

    // should work on an already sorted list
    rc = List_bubble_sort(words, (List_compare) strcmp);
    mu_assert(rc == 0, "Bubble sort of already sorted failed.");
    mu_assert(is_sorted(words),
            "Words should be sort if already bubble sorted.");

    List_destroy(words);

    // should work on an empty list
    words = List_create(words);
    rc = List_bubble_sort(words, (List_compare) strcmp);
    mu_assert(rc == 0, "Bubble sort failed on empty list.");
    mu_assert(is_sorted(words), "Words should be sorted if empty.");

    List_destroy(words);

    return NULL;
}

char *test_merge_sort()
{
    List *words = create_words();

    // should work on a list that needs sorting
    List *res = List_merge_sort(words, (List_compare) strcmp);
    mu_assert(is_sorted(res), "Words are not sorted after merge sort.");

    List *res2 = List_merge_sort(res, (List_compare) strcmp);
    mu_assert(is_sorted(res),
            "Should still be sorted after merge sort.");
    List_destroy(res2);
    List_destroy(res);

    List_destroy(words);
    return NULL;
}

char *all_tests()
{
    mu_suite_start();

    mu_run_test(test_bubble_sort);
    mu_run_test(test_merge_sort);

    return NULL;
}

RUN_TESTS(all_tests);

```

You should be able to create this and figure out how it works.

I will assume you've done that, and now to code review.

Bubble Sort

Code review of bubble sort.

Start with the unit test and move from there.

Merge Sort

Code review of merge sort.

Improving It

* The merge sort does a crazy amount of copying and creating lists, so find ways to reduce this.
* The bubble sort description in Wikipedia mentions a few optimizations. Try to implement them.
* Can you use the ``List_split`` and ``List_join`` (if you implemented them) to improve merge sort?
* Go through of all the defensive programming checks and improve the robustness of
  this implementation, protecting against bad ``NULL`` pointers, and then create
  an optional debug level invariant that works like ``is_sorted`` does
  after a sort.

Breaking It

* Overload the data structure to hit the worst case time complexity.
* Give it a bad data structure.

Extra Credit

* Create a unit test that compares the performance of the two algorithms.  You'll want to look at ``man 3 time`` for a basic timer function,
  and run enough iterations to at least have a few seconds of samples.
* Play with the amount of data in the lists that need to be sorted and see if that changes your timing.
* Find a way to simulate filling different sized random lists, measuring how long they take. Then, graph the result to see how it compares to the
  description of the algorithm.

Extra Credit

* Try to explain why sorting linked lists is a really bad idea.
* Implement a ``List_insert_sorted`` that will take a given value, and using the ``List_compare``, insert the element at the
  right position so that the list is always sorted.  How does using this method compare to sorting a list after you've built it?
* Try implementing the bottom-up merge sort described on the Wikipedia page.  The code there is already C, so it should be easy to
  recreate, but try to understand how it's working compared to the slower one I have here.

### Exercise 34 Dynamic Array

The Plan

Learn about dynamic arrays, a very useful datastructure.

Code Review

.\ex34\darray.h

```c
#ifndef _DArray_h
#define _DArray_h
#include <stdlib.h>
#include <assert.h>
#include <lcthw/dbg.h>

typedef struct DArray {
    int end;
    int max;
    size_t element_size;
    size_t expand_rate;
    void **contents;
} DArray;

DArray *DArray_create(size_t element_size, size_t initial_max);

void DArray_destroy(DArray * array);

void DArray_clear(DArray * array);

int DArray_expand(DArray * array);

int DArray_contract(DArray * array);

int DArray_push(DArray * array, void *el);

void *DArray_pop(DArray * array);

void DArray_clear_destroy(DArray * array);

#define DArray_last(A) ((A)->contents[(A)->end - 1])
#define DArray_first(A) ((A)->contents[0])
#define DArray_end(A) ((A)->end)
#define DArray_count(A) DArray_end(A)
#define DArray_max(A) ((A)->max)

#define DEFAULT_EXPAND_RATE 300

static inline void DArray_set(DArray * array, int i, void *el)
{
    check(i < array->max, "darray attempt to set past max");
    if (i > array->end)
        array->end = i;
    array->contents[i] = el;
error:
    return;
}

static inline void *DArray_get(DArray * array, int i)
{
    check(i < array->max, "darray attempt to get past max");
    return array->contents[i];
error:
    return NULL;
}

static inline void *DArray_remove(DArray * array, int i)
{
    void *el = array->contents[i];

    array->contents[i] = NULL;

    return el;
}

static inline void *DArray_new(DArray * array)
{
    check(array->element_size > 0,
            "Can't use DArray_new on 0 size darrays.");

    return calloc(1, array->element_size);

error:
    return NULL;
}

#define DArray_free(E) free((E))

#endif

```

.\ex34\darray.c

```c
#include <lcthw/darray.h>
#include <assert.h>

DArray *DArray_create(size_t element_size, size_t initial_max)
{
    DArray *array = malloc(sizeof(DArray));
    check_mem(array);
    array->max = initial_max;
    check(array->max > 0, "You must set an initial_max > 0.");

    array->contents = calloc(initial_max, sizeof(void *));
    check_mem(array->contents);

    array->end = 0;
    array->element_size = element_size;
    array->expand_rate = DEFAULT_EXPAND_RATE;

    return array;

error:
    if (array)
        free(array);
    return NULL;
}

void DArray_clear(DArray * array)
{
    int i = 0;
    if (array->element_size > 0) {
        for (i = 0; i < array->max; i++) {
            if (array->contents[i] != NULL) {
                free(array->contents[i]);
            }
        }
    }
}

static inline int DArray_resize(DArray * array, size_t newsize)
{
    array->max = newsize;
    check(array->max > 0, "The newsize must be > 0.");

    void *contents = realloc(
            array->contents, array->max * sizeof(void *));
    // check contents and assume realloc doesn't harm the original on error

    check_mem(contents);

    array->contents = contents;

    return 0;
error:
    return -1;
}

int DArray_expand(DArray * array)
{
    size_t old_max = array->max;
    check(DArray_resize(array, array->max + array->expand_rate) == 0,
            "Failed to expand array to new size: %d",
            array->max + (int)array->expand_rate);

    memset(array->contents + old_max, 0, array->expand_rate + 1);
    return 0;

error:
    return -1;
}

int DArray_contract(DArray * array)
{
    int new_size = array->end < (int)array->expand_rate ? 
            (int)array->expand_rate : array->end;

    return DArray_resize(array, new_size + 1);
}

void DArray_destroy(DArray * array)
{
    if (array) {
        if (array->contents)
            free(array->contents);
        free(array);
    }
}

void DArray_clear_destroy(DArray * array)
{
    DArray_clear(array);
    DArray_destroy(array);
}

int DArray_push(DArray * array, void *el)
{
    array->contents[array->end] = el;
    array->end++;

    if (DArray_end(array) >= DArray_max(array)) {
        return DArray_expand(array);
    } else {
        return 0;
    }
}

void *DArray_pop(DArray * array)
{
    check(array->end - 1 >= 0, "Attempt to pop from empty array.");

    void *el = DArray_remove(array, array->end - 1);
    array->end--;

    if (DArray_end(array) > (int)array->expand_rate
            && DArray_end(array) % array->expand_rate) {
        DArray_contract(array);
    }

    return el;
error:
    return NULL;
}

```

.\ex34\darray_tests.c

```c
#include "minunit.h"
#include <lcthw/darray.h>

static DArray *array = NULL;
static int *val1 = NULL;
static int *val2 = NULL;

char *test_create()
{
    array = DArray_create(sizeof(int), 100);
    mu_assert(array != NULL, "DArray_create failed.");
    mu_assert(array->contents != NULL, "contents are wrong in darray");
    mu_assert(array->end == 0, "end isn't at the right spot");
    mu_assert(array->element_size == sizeof(int),
            "element size is wrong.");
    mu_assert(array->max == 100, "wrong max length on initial size");

    return NULL;
}

char *test_destroy()
{
    DArray_destroy(array);

    return NULL;
}

char *test_new()
{
    val1 = DArray_new(array);
    mu_assert(val1 != NULL, "failed to make a new element");

    val2 = DArray_new(array);
    mu_assert(val2 != NULL, "failed to make a new element");

    return NULL;
}

char *test_set()
{
    DArray_set(array, 0, val1);
    DArray_set(array, 1, val2);

    return NULL;
}

char *test_get()
{
    mu_assert(DArray_get(array, 0) == val1, "Wrong first value.");
    mu_assert(DArray_get(array, 1) == val2, "Wrong second value.");

    return NULL;
}

char *test_remove()
{
    int *val_check = DArray_remove(array, 0);
    mu_assert(val_check != NULL, "Should not get NULL.");
    mu_assert(*val_check == *val1, "Should get the first value.");
    mu_assert(DArray_get(array, 0) == NULL, "Should be gone.");
    DArray_free(val_check);

    val_check = DArray_remove(array, 1);
    mu_assert(val_check != NULL, "Should not get NULL.");
    mu_assert(*val_check == *val2, "Should get the first value.");
    mu_assert(DArray_get(array, 1) == NULL, "Should be gone.");
    DArray_free(val_check);

    return NULL;
}

char *test_expand_contract()
{
    int old_max = array->max;
    DArray_expand(array);
    mu_assert((unsigned int)array->max == old_max + array->expand_rate,
            "Wrong size after expand.");

    DArray_contract(array);
    mu_assert((unsigned int)array->max == array->expand_rate + 1,
            "Should stay at the expand_rate at least.");

    DArray_contract(array);
    mu_assert((unsigned int)array->max == array->expand_rate + 1,
            "Should stay at the expand_rate at least.");

    return NULL;
}

char *test_push_pop()
{
    int i = 0;
    for (i = 0; i < 1000; i++) {
        int *val = DArray_new(array);
        *val = i * 333;
        DArray_push(array, val);
    }

    mu_assert(array->max == 1201, "Wrong max size.");

    for (i = 999; i >= 0; i--) {
        int *val = DArray_pop(array);
        mu_assert(val != NULL, "Shouldn't get a NULL.");
        mu_assert(*val == i * 333, "Wrong value.");
        DArray_free(val);
    }

    return NULL;
}

char *all_tests()
{
    mu_suite_start();

    mu_run_test(test_create);
    mu_run_test(test_new);
    mu_run_test(test_set);
    mu_run_test(test_get);
    mu_run_test(test_remove);
    mu_run_test(test_expand_contract);
    mu_run_test(test_push_pop);
    mu_run_test(test_destroy);

    return NULL;
}

RUN_TESTS(all_tests);

```

Starting with the header file to implement, then the test, then the implementation.

The Analysis

DArray Advantages

* Iteration:  You can just use a basic for-loop and ``DArray_count``
  with ``DArray_get``, and you're done.  No special macros needed, and
  it's faster because you aren't walking through pointers.
* Indexing:  You can use ``DArray_get`` and ``DArray_set`` to
  access any element at random, but with a ``List`` you have to go
  through N elements to get to N+1.
* Destroying:  You can just free the struct and the ``contents`` in
  two operations.  A ``List`` requires a series of ``free`` calls
  and walking every element.

DArray Advantages

* Cloning: You can also clone it in just two operations (plus whatever
  it's storing) by copying the struct and ``contents``.  A list
  again requires walking through the whole thing and copying every ``ListNode``
  plus its value.
* Sorting: As you saw, ``List`` is horrible if you need to keep the
  data sorted.  A ``DArray`` opens up a whole class of great sorting
  algorithms, because now you can access elements randomly.
* Large Data: If you need to keep around a lot of data, then a ``DArray``
  wins since its base, ``contents``, takes up less memory than the same
  number of ``ListNode`` structs.

DArray Disadvantages

* Insert and remove on the front (what I called shift).  A ``DArray``
  needs special treatment to be able to do this efficiently, and usually it
  has to do some copying.
* Splitting or joining:  A ``List`` can just copy some pointers and
  it's done, but with a ``DArray``, you have copy all of the
  arrays involved.

DArray Disadvantages

* Small Data. If you only need to store a few elements, then typically the
  storage will be less in a ``List`` than a generic ``DArray``. This is because
  the ``DArray`` needs to expand the backing store to accommodate future
  inserts, while a ``List`` only makes what it needs.

Breaking It

* Forget to check the return value from malloc and then use the buffer.
* Getting the end and start count of the buffer wrong.  Easy to do an off-by-one here.
* Exploit the insert and delete costs to cause a denial of service.

Extra Credit

* Improve the unit tests to cover more of the operations, and test them
  using a for-loop to ensure that they work.
* Research what it would take to implement bubble sort and merge sort
  for DArray, but don't do it yet.  I'll be implementing DArray algorithms
  next, so you'll do this then.

Extra Credit

* Write some performance tests for common operations and compare them
  to the same operations in ``List``.  You did some of this already, but this
  time, write a unit test that repeatedly does the operation in question, and
  then in the main runner, do the timing.

Extra Credit

* Look at how the ``DArray_expand`` is implemented using a constant increase (size + 300).
  Typically, dynamic arrays are implemented with a multiplicative increase (size * 2), but I've
  found this to cost needless memory for no real performance gain.  Test my assertion
  and see when you'd want a multiplicative increase instead of a constant increase.

### Exercise 35 Sorting and Searching

The Plan

* Make a simple DArray sorting library using existing functions.
* Implement a new structure and algorithm called a "Radix Map".
* Create a binary search algorithm for the RadixMap.

The DArray Code

.\ex35\darray_algos.h

```c
#ifndef darray_algos_h
#define darray_algos_h

#include <lcthw/darray.h>

typedef int (*DArray_compare) (const void *a, const void *b);

int DArray_qsort(DArray * array, DArray_compare cmp);

int DArray_heapsort(DArray * array, DArray_compare cmp);

int DArray_mergesort(DArray * array, DArray_compare cmp);

#endif

```

.\ex35\darray_algos.c

```c
#include <lcthw/darray_algos.h>
#include <stdlib.h>

int DArray_qsort(DArray * array, DArray_compare cmp)
{
    qsort(array->contents, DArray_count(array), sizeof(void *), cmp);
    return 0;
}

int DArray_heapsort(DArray * array, DArray_compare cmp)
{
    return heapsort(array->contents, DArray_count(array),
            sizeof(void *), cmp);
}

int DArray_mergesort(DArray * array, DArray_compare cmp)
{
    return mergesort(array->contents, DArray_count(array),
            sizeof(void *), cmp);
}

```

.\ex35\darray_algos_tests.c

```c
#include "minunit.h"
#include <lcthw/darray_algos.h>

int testcmp(char **a, char **b)
{
    return strcmp(*a, *b);
}

DArray *create_words()
{
    DArray *result = DArray_create(0, 5);
    char *words[] = { "asdfasfd",
        "werwar", "13234", "asdfasfd", "oioj" };
    int i = 0;

    for (i = 0; i < 5; i++) {
        DArray_push(result, words[i]);
    }

    return result;
}

int is_sorted(DArray * array)
{
    int i = 0;

    for (i = 0; i < DArray_count(array) - 1; i++) {
        if (strcmp(DArray_get(array, i), DArray_get(array, i + 1)) > 0) {
            return 0;
        }
    }

    return 1;
}

char *run_sort_test(int (*func) (DArray *, DArray_compare),
        const char *name)
{
    DArray *words = create_words();
    mu_assert(!is_sorted(words), "Words should start not sorted.");

    debug("--- Testing %s sorting algorithm", name);
    int rc = func(words, (DArray_compare) testcmp);
    mu_assert(rc == 0, "sort failed");
    mu_assert(is_sorted(words), "didn't sort it");

    DArray_destroy(words);

    return NULL;
}

char *test_qsort()
{
    return run_sort_test(DArray_qsort, "qsort");
}

char *test_heapsort()
{
    return run_sort_test(DArray_heapsort, "heapsort");
}

char *test_mergesort()
{
    return run_sort_test(DArray_mergesort, "mergesort");
}

char *all_tests()
{
    mu_suite_start();

    mu_run_test(test_qsort);
    mu_run_test(test_heapsort);
    mu_run_test(test_mergesort);

    return NULL;
}

RUN_TESTS(all_tests);

```

Continuing the code review method with a part of DArray.

The RadixMap Code

.\ex35\radixmap.h

```c
#ifndef _radixmap_h
#include <stdint.h>

typedef union RMElement {
    uint64_t raw;
    struct {
        uint32_t key;
        uint32_t value;
    } data;
} RMElement;

typedef struct RadixMap {
    size_t max;
    size_t end;
    uint32_t counter;
    RMElement *contents;
    RMElement *temp;
} RadixMap;

RadixMap *RadixMap_create(size_t max);

void RadixMap_destroy(RadixMap * map);

void RadixMap_sort(RadixMap * map);

RMElement *RadixMap_find(RadixMap * map, uint32_t key);

int RadixMap_add(RadixMap * map, uint32_t key, uint32_t value);

int RadixMap_delete(RadixMap * map, RMElement * el);

#endif

```

.\ex35\radixmap.c

```c
/*
 * Based on code by <NAME> then heavily modified by <NAME>.
 */

#include <stdio.h>
#include <stdlib.h>
#include <assert.h>
#include <lcthw/radixmap.h>
#include <lcthw/dbg.h>

RadixMap *RadixMap_create(size_t max)
{
    RadixMap *map = calloc(sizeof(RadixMap), 1);
    check_mem(map);

    map->contents = calloc(sizeof(RMElement), max + 1);
    check_mem(map->contents);

    map->temp = calloc(sizeof(RMElement), max + 1);
    check_mem(map->temp);

    map->max = max;
    map->end = 0;

    return map;
error:
    return NULL;
}

void RadixMap_destroy(RadixMap * map)
{
    if (map) {
        free(map->contents);
        free(map->temp);
        free(map);
    }
}

#define ByteOf(x,y) (((uint8_t *)x)[(y)])

static inline void radix_sort(short offset, uint64_t max,
        uint64_t * source, uint64_t * dest)
{
    uint64_t count[256] = { 0 };
    uint64_t *cp = NULL;
    uint64_t *sp = NULL;
    uint64_t *end = NULL;
    uint64_t s = 0;
    uint64_t c = 0;

    // count occurences of every byte value
    for (sp = source, end = source + max; sp < end; sp++) {
        count[ByteOf(sp, offset)]++;
    }

    // transform count into index by summing
    // elements and storing into same array
    for (s = 0, cp = count, end = count + 256; cp < end; cp++) {
        c = *cp;
        *cp = s;
        s += c;
    }

    // fill dest with the right values in the right place
    for (sp = source, end = source + max; sp < end; sp++) {
        cp = count + ByteOf(sp, offset);
        dest[*cp] = *sp;
        ++(*cp);
    }
}

void RadixMap_sort(RadixMap * map)
{
    uint64_t *source = &map->contents[0].raw;
    uint64_t *temp = &map->temp[0].raw;

    radix_sort(0, map->end, source, temp);
    radix_sort(1, map->end, temp, source);
    radix_sort(2, map->end, source, temp);
    radix_sort(3, map->end, temp, source);
}

RMElement *RadixMap_find(RadixMap * map, uint32_t to_find)
{
    int low = 0;
    int high = map->end - 1;
    RMElement *data = map->contents;

    while (low <= high) {
        int middle = low + (high - low) / 2;
        uint32_t key = data[middle].data.key;

        if (to_find < key) {
            high = middle - 1;
        } else if (to_find > key) {
            low = middle + 1;
        } else {
            return &data[middle];
        }
    }

    return NULL;
}

int RadixMap_add(RadixMap * map, uint32_t key, uint32_t value)
{
    check(key < UINT32_MAX, "Key can't be equal to UINT32_MAX.");

    RMElement element = {.data = {.key = key,.value = value} };
    check(map->end + 1 < map->max, "RadixMap is full.");

    map->contents[map->end++] = element;

    RadixMap_sort(map);

    return 0;

error:
    return -1;
}

int RadixMap_delete(RadixMap * map, RMElement * el)
{
    check(map->end > 0, "There is nothing to delete.");
    check(el != NULL, "Can't delete a NULL element.");

    el->data.key = UINT32_MAX;

    if (map->end > 1) {
        // don't bother resorting a map of 1 length
        RadixMap_sort(map);
    }

    map->end--;

    return 0;
error:
    return -1;
}

```

.\ex35\radixmap_tests.c

```c
#include "minunit.h"
#include <lcthw/radixmap.h>
#include <time.h>

static int make_random(RadixMap * map)
{
    size_t i = 0;

    for (i = 0; i < map->max - 1; i++) {
        uint32_t key = (uint32_t) (rand() | (rand() << 16));
        check(RadixMap_add(map, key, i) == 0, "Failed to add key %u.",
                key);
    }

    return i;

error:
    return 0;
}

static int check_order(RadixMap * map)
{
    RMElement d1, d2;
    unsigned int i = 0;

    // only signal errors if any (should not be)
    for (i = 0; map->end > 0 && i < map->end - 1; i++) {
        d1 = map->contents[i];
        d2 = map->contents[i + 1];

        if (d1.data.key > d2.data.key) {
            debug("FAIL:i=%u, key: %u, value: %u, equals max? %d\n", i,
                    d1.data.key, d1.data.value,
                    d2.data.key == UINT32_MAX);
            return 0;
        }
    }

    return 1;
}

static int test_search(RadixMap * map)
{
    unsigned i = 0;
    RMElement *d = NULL;
    RMElement *found = NULL;

    for (i = map->end / 2; i < map->end; i++) {
        d = &map->contents[i];
        found = RadixMap_find(map, d->data.key);
        check(found != NULL, "Didn't find %u at %u.", d->data.key, i);
        check(found->data.key == d->data.key,
                "Got the wrong result: %p:%u looking for %u at %u", found,
                found->data.key, d->data.key, i);
    }

    return 1;
error:
    return 0;
}

// test for big number of elements
static char *test_operations()
{
    size_t N = 200;

    RadixMap *map = RadixMap_create(N);
    mu_assert(map != NULL, "Failed to make the map.");
    mu_assert(make_random(map), "Didn't make a random fake radix map.");

    RadixMap_sort(map);
    mu_assert(check_order(map),
            "Failed to properly sort the RadixMap.");

    mu_assert(test_search(map), "Failed the search test.");
    mu_assert(check_order(map),
            "RadixMap didn't stay sorted after search.");

    while (map->end > 0) {
        RMElement *el = RadixMap_find(map,
                map->contents[map->end / 2].data.key);
        mu_assert(el != NULL, "Should get a result.");

        size_t old_end = map->end;

        mu_assert(RadixMap_delete(map, el) == 0, "Didn't delete it.");
        mu_assert(old_end - 1 == map->end, "Wrong size after delete.");

        // test that the end is now the old value,
        // but uint32 max so it trails off
        mu_assert(check_order(map),
                "RadixMap didn't stay sorted after delete.");
    }

    RadixMap_destroy(map);

    return NULL;
}

char *all_tests()
{
    mu_suite_start();
    srand(time(NULL));

    mu_run_test(test_operations);

    return NULL;
}

RUN_TESTS(all_tests);

```

Code review this code next.

The Binary Search Code

Finally, code review of the BSTree code.

Improving It

* Use a binary search to find the minimum position for the
  new element, then only sort from there to the end.  You find the
  minimum, put the new element on the end, and then just sort from
  the minimum on.   This will cut your sort space down
  considerably most of the time.
* Keep track of the biggest key currently being used, and then only
  sort enough digits to handle that key.  You can also keep track
  of the smallest number, and then only sort the digits necessary
  for the range.  To do this, you'll have to start caring about
  CPU integer ordering (endianness).

Extra Credit

* Implement quicksort, heapsort, and merge sort and then provide a *#define*
  that lets you pick between the two, or create a second set of functions
  you can call.  Use the technique I taught you to read the Wikipedia page
  for the algorithm, and then implement it with the psuedo-code.
* Compare the performance of your optimizations to the original implementations.

Extra Credit

* Use these sorting functions to create a *DArray_sort_add* that
  adds elements to the *DArray* but sorts the array after.
* Write a *DArray_find* that uses the binary search algorithm from
  *RadixMap_find* and the *DArray_compare* to find elements
  in a sorted *DArray*.

### Exercise 36 Safer Strings
.\ex36\ex36.c

```c
void copy(char to[], char from[])
{
    int i = 0;

    // while loop will not end if from isn't '\0' terminated
    while ((to[i] = from[i]) != '\0') {
        ++i;
    }
}

int safercopy(int from_len, char *from, int to_len, char *to)
{
    int i = 0;
    int max = from_len > to_len - 1 ? to_len - 1 : from_len;

    // to_len must have at least 1 byte
    if (from_len < 0 || to_len <= 0)
        return -1;

    for (i = 0; i < max; i++) {
        to[i] = from[i];
    }

    to[to_len - 1] = '\0';

    return i;
}

```

.\ex36\ex36.c

```c
void copy(char to[], char from[])
{
    int i = 0;

    // while loop will not end if from isn't '\0' terminated
    while ((to[i] = from[i]) != '\0') {
        ++i;
    }
}

int safercopy(int from_len, char *from, int to_len, char *to)
{
    int i = 0;
    int max = from_len > to_len - 1 ? to_len - 1 : from_len;

    // to_len must have at least 1 byte
    if (from_len < 0 || to_len <= 0)
        return -1;

    for (i = 0; i < max; i++) {
        to[i] = from[i];
    }

    to[to_len - 1] = '\0';

    return i;
}

```

The Plan

Learn about an alternative string implementation to avoid most C string problems.

C Strings Suck

It is impossible to safely process strings in C.

The bstrling Library

An alternative is a library that provides alternative APIs for working with
C strings.

The Common Functions

    bfromcstr    Create a bstring from a C style constant.
    blk2bstr     Do the same thing, but give the length of the buffer.
    bstrcpy      Copy a bstring.
    bassign      Set one bstring to another.
    bassigncstr  Set a bstring to a C string's contents.
    bassignblk   Set a bstring to a C string but give the length.
    bdestroy     Destroy a bstring.
    bconcat      Concatenate one bstring onto another.
    bstricmp     Compare two bstrings returning the same result as strcmp.
    biseq        Tests if two bstrings are equal.

The Common Functions

    binstr       Tells if one bstring is in another.
    bfindreplace Find one bstring in another, then replace it with a third.
    bsplit       Split a bstring into a bstrList.
    bformat      Do a format string, which is super handy.
    blength      Get the length of a bstring.
    bdata        Get the data from a bstring.
    bchar        Get a char from a bstring.

Extra Credit

There is only one extra credit and that's to write a *tests/bstr_tests.c* file that
tests all of these functions.  The bstrlib comes with a test that you can reference it if needed.

### Exercise 37 Hashmaps

The Plan

Implement a Hashmap in C.
In Python these are called Dictionaries.

Hashmaps Visually

Hashmaps are very intuitive once you know how a DArray works.
It's all about the hashing function used.

Code Review

.\ex37\hashmap.h

```c
#ifndef _lcthw_Hashmap_h
#define _lcthw_Hashmap_h

#include <stdint.h>
#include <lcthw/darray.h>

#define DEFAULT_NUMBER_OF_BUCKETS 100

typedef int (*Hashmap_compare) (void *a, void *b);
typedef uint32_t(*Hashmap_hash) (void *key);

typedef struct Hashmap {
    DArray *buckets;
    Hashmap_compare compare;
    Hashmap_hash hash;
} Hashmap;

typedef struct HashmapNode {
    void *key;
    void *data;
    uint32_t hash;
} HashmapNode;

typedef int (*Hashmap_traverse_cb) (HashmapNode * node);

Hashmap *Hashmap_create(Hashmap_compare compare, Hashmap_hash);
void Hashmap_destroy(Hashmap * map);

int Hashmap_set(Hashmap * map, void *key, void *data);
void *Hashmap_get(Hashmap * map, void *key);

int Hashmap_traverse(Hashmap * map, Hashmap_traverse_cb traverse_cb);

void *Hashmap_delete(Hashmap * map, void *key);

#endif

```

.\ex37\hashmap.c

```c
#undef NDEBUG
#include <stdint.h>
#include <lcthw/hashmap.h>
#include <lcthw/dbg.h>
#include <lcthw/bstrlib.h>

static int default_compare(void *a, void *b)
{
    return bstrcmp((bstring) a, (bstring) b);
}

/** 
 * Simple Bob Jenkins's hash algorithm taken from the
 * wikipedia description.
 */
static uint32_t default_hash(void *a)
{
    size_t len = blength((bstring) a);
    char *key = bdata((bstring) a);
    uint32_t hash = 0;
    uint32_t i = 0;

    for (hash = i = 0; i < len; ++i) {
        hash += key[i];
        hash += (hash << 10);
        hash ^= (hash >> 6);
    }

    hash += (hash << 3);
    hash ^= (hash >> 11);
    hash += (hash << 15);

    return hash;
}

Hashmap *Hashmap_create(Hashmap_compare compare, Hashmap_hash hash)
{
    Hashmap *map = calloc(1, sizeof(Hashmap));
    check_mem(map);

    map->compare = compare == NULL ? default_compare : compare;
    map->hash = hash == NULL ? default_hash : hash;
    map->buckets = DArray_create(
            sizeof(DArray *), DEFAULT_NUMBER_OF_BUCKETS);
    map->buckets->end = map->buckets->max;    // fake out expanding it
    check_mem(map->buckets);

    return map;

error:
    if (map) {
        Hashmap_destroy(map);
    }

    return NULL;
}

void Hashmap_destroy(Hashmap * map)
{
    int i = 0;
    int j = 0;

    if (map) {
        if (map->buckets) {
            for (i = 0; i < DArray_count(map->buckets); i++) {
                DArray *bucket = DArray_get(map->buckets, i);
                if (bucket) {
                    for (j = 0; j < DArray_count(bucket); j++) {
                        free(DArray_get(bucket, j));
                    }
                    DArray_destroy(bucket);
                }
            }
            DArray_destroy(map->buckets);
        }

        free(map);
    }
}

static inline HashmapNode *Hashmap_node_create(int hash, void *key,
        void *data)
{
    HashmapNode *node = calloc(1, sizeof(HashmapNode));
    check_mem(node);

    node->key = key;
    node->data = data;
    node->hash = hash;

    return node;

error:
    return NULL;
}

static inline DArray *Hashmap_find_bucket(Hashmap * map, void *key,
        int create,
        uint32_t * hash_out)
{
    uint32_t hash = map->hash(key);
    int bucket_n = hash % DEFAULT_NUMBER_OF_BUCKETS;
    check(bucket_n >= 0, "Invalid bucket found: %d", bucket_n);
    // store it for the return so the caller can use it
    *hash_out = hash;

    DArray *bucket = DArray_get(map->buckets, bucket_n);

    if (!bucket && create) {
        // new bucket, set it up
        bucket = DArray_create(
                sizeof(void *), DEFAULT_NUMBER_OF_BUCKETS);
        check_mem(bucket);
        DArray_set(map->buckets, bucket_n, bucket);
    }

    return bucket;

error:
    return NULL;
}

int Hashmap_set(Hashmap * map, void *key, void *data)
{
    uint32_t hash = 0;
    DArray *bucket = Hashmap_find_bucket(map, key, 1, &hash);
    check(bucket, "Error can't create bucket.");

    HashmapNode *node = Hashmap_node_create(hash, key, data);
    check_mem(node);

    DArray_push(bucket, node);

    return 0;

error:
    return -1;
}

static inline int Hashmap_get_node(Hashmap * map, uint32_t hash,
        DArray * bucket, void *key)
{
    int i = 0;

    for (i = 0; i < DArray_end(bucket); i++) {
        debug("TRY: %d", i);
        HashmapNode *node = DArray_get(bucket, i);
        if (node->hash == hash && map->compare(node->key, key) == 0) {
            return i;
        }
    }

    return -1;
}

void *Hashmap_get(Hashmap * map, void *key)
{
    uint32_t hash = 0;
    DArray *bucket = Hashmap_find_bucket(map, key, 0, &hash);
    if (!bucket) return NULL;

    int i = Hashmap_get_node(map, hash, bucket, key);
    if (i == -1) return NULL;

    HashmapNode *node = DArray_get(bucket, i);
    check(node != NULL,
            "Failed to get node from bucket when it should exist.");

    return node->data;

error:            // fallthrough
    return NULL;
}

int Hashmap_traverse(Hashmap * map, Hashmap_traverse_cb traverse_cb)
{
    int i = 0;
    int j = 0;
    int rc = 0;

    for (i = 0; i < DArray_count(map->buckets); i++) {
        DArray *bucket = DArray_get(map->buckets, i);
        if (bucket) {
            for (j = 0; j < DArray_count(bucket); j++) {
                HashmapNode *node = DArray_get(bucket, j);
                rc = traverse_cb(node);
                if (rc != 0)
                    return rc;
            }
        }
    }

    return 0;
}

void *Hashmap_delete(Hashmap * map, void *key)
{
    uint32_t hash = 0;
    DArray *bucket = Hashmap_find_bucket(map, key, 0, &hash);
    if (!bucket)
        return NULL;

    int i = Hashmap_get_node(map, hash, bucket, key);
    if (i == -1)
        return NULL;

    HashmapNode *node = DArray_get(bucket, i);
    void *data = node->data;
    free(node);

    HashmapNode *ending = DArray_pop(bucket);

    if (ending != node) {
        // alright looks like it's not the last one, swap it
        DArray_set(bucket, i, ending);
    }

    return data;
}

```

Conducting a review of Hashmap by following the test.

Improving It

* You can use a sort on each bucket so that they're always sorted.
  This increases your insert time but decreases your find time, because
  you can then use a binary search to find each node.  Right now,
  it's looping through all of the nodes in a bucket just to find one.
* You can dynamically size the number of buckets, or let the caller
  specify the number for each *Hashmap* created.
* You can use a better *default_hash*.  There are tons of them.

Improving It

* This (and nearly every *Hashmap*) is vulnerable to someone picking
  keys that will fill only one bucket, and then tricking your program
  into processing them.  This then makes your program run slower because
  it changes from processing a *Hashmap* to effectively processing
  a single *DArray*.  If you sort the nodes in the bucket, this
  helps, but you can also use better hashing functions, and for
  the really paranoid programmer, add a random salt so that keys can't be predicted.

Improving It

* You could have it delete buckets that are empty of nodes to save space,
  or put empty buckets into a cache so you can save on time lost creating and destroying
  them.
* Right now, it just adds elements even if they already exist.  Write an
  alternative set method that only adds an element if it isn't set already.

Extra Credit

* Research the *Hashmap* implementation in your favorite programming language to see what features it has.
* Find out what the major disadvantages of a *Hashmap* are and how to avoid them.  For example, it doesn't preserve order without special changes, nor does it work when you need to find things based on parts
  of keys.
* Write a unit test that demonstrates the defect of filling a *Hashmap* with keys that land
  in the same bucket, then test how this impacts performance.  A good way to do this is to just reduce
  the number of buckets to something stupid, like five.

### Exercise 38 Hashmap Algorithms

The Plan

Learn three different string hashing algorithms and make them dynamically available
to the Hashmap.

Code Review

.\ex38\hashmap_algos.h

```c
#ifndef hashmap_algos_h
#define hashmap_algos_h

#include <stdint.h>

uint32_t Hashmap_fnv1a_hash(void *data);

uint32_t Hashmap_adler32_hash(void *data);

uint32_t Hashmap_djb_hash(void *data);

#endif

```

.\ex38\hashmap_algos.c

```c
#include <lcthw/hashmap_algos.h>
#include <lcthw/bstrlib.h>

// settings taken from
// http://www.isthe.com/chongo/tech/comp/fnv/index.html#FNV-param
const uint32_t FNV_PRIME = 16777619;
const uint32_t FNV_OFFSET_BASIS = 2166136261;

uint32_t Hashmap_fnv1a_hash(void *data)
{
    bstring s = (bstring) data;
    uint32_t hash = FNV_OFFSET_BASIS;
    int i = 0;

    for (i = 0; i < blength(s); i++) {
        hash ^= bchare(s, i, 0);
        hash *= FNV_PRIME;
    }

    return hash;
}

const int MOD_ADLER = 65521;

uint32_t Hashmap_adler32_hash(void *data)
{
    bstring s = (bstring) data;
    uint32_t a = 1, b = 0;
    int i = 0;

    for (i = 0; i < blength(s); i++) {
        a = (a + bchare(s, i, 0)) % MOD_ADLER;
        b = (b + a) % MOD_ADLER;
    }

    return (b << 16) | a;
}

uint32_t Hashmap_djb_hash(void *data)
{
    bstring s = (bstring) data;
    uint32_t hash = 5381;
    int i = 0;

    for (i = 0; i < blength(s); i++) {
        hash = ((hash << 5) + hash) + bchare(s, i, 0);    /* hash * 33 + c */
    }

    return hash;
}

```

The default is the Jenkin's hash.

You added the FNV1a, Adler32, and DJB hashing algorithms.

Review the code for FNV1a vs. DJB.

Breaking It

In this exercise you will attempt to write the worst hashing function that can
pass for a real one.  Try to make one that either looks complicated but
statistically is way off, or is a discrete change to an existing one that is a
bad change.

Extra Credit

* Take the ``default_hash`` out of the ``hashmap.c``, make it
  one of the algorithms in ``hashmap_algos.c``, and then make all
  of the tests work again.
* Add the ``default_hash`` to the ``hashmap_algos_tests.c``
  test and compare its statistics to the other hash functions.
* Find a few more hash functions and add them, too. You can never have too
  many hash functions!

### Exercise 39 String Algorithms

The Plan

Develop a formal code review procedure.

The Code

.\ex39\string_algos.h

```c
#ifndef string_algos_h
#define string_algos_h

#include <lcthw/bstrlib.h>
#include <lcthw/darray.h>

typedef struct StringScanner {
    bstring in;
    const unsigned char *haystack;
    ssize_t hlen;
    const unsigned char *needle;
    ssize_t nlen;
    size_t skip_chars[UCHAR_MAX + 1];
} StringScanner;

int String_find(bstring in, bstring what);

StringScanner *StringScanner_create(bstring in);

int StringScanner_scan(StringScanner * scan, bstring tofind);

void StringScanner_destroy(StringScanner * scan);

#endif

```

.\ex39\string_algos.c

```c
#include <lcthw/string_algos.h>
#include <limits.h>

static inline void String_setup_skip_chars(size_t * skip_chars,
        const unsigned char *needle,
        ssize_t nlen)
{
    size_t i = 0;
    size_t last = nlen - 1;

    for (i = 0; i < UCHAR_MAX + 1; i++) {
        skip_chars[i] = nlen;
    }

    for (i = 0; i < last; i++) {
        skip_chars[needle[i]] = last - i;
    }
}

static inline const unsigned char *String_base_search(const unsigned
        char *haystack,
        ssize_t hlen,
        const unsigned
        char *needle,
        ssize_t nlen,
        size_t *
        skip_chars)
{
    size_t i = 0;
    size_t last = nlen - 1;

    assert(haystack != NULL && "Given bad haystack to search.");
    assert(needle != NULL && "Given bad needle to search for.");

    check(nlen > 0, "nlen can't be <= 0");
    check(hlen > 0, "hlen can't be <= 0");

    while (hlen >= nlen) {
        for (i = last; haystack[i] == needle[i]; i--) {
            if (i == 0) {
                return haystack;
            }
        }

        hlen -= skip_chars[haystack[last]];
        haystack += skip_chars[haystack[last]];
    }

error:            // fallthrough
    return NULL;
}

int String_find(bstring in, bstring what)
{
    const unsigned char *found = NULL;

    const unsigned char *haystack = (const unsigned char *)bdata(in);
    ssize_t hlen = blength(in);
    const unsigned char *needle = (const unsigned char *)bdata(what);
    ssize_t nlen = blength(what);
    size_t skip_chars[UCHAR_MAX + 1] = { 0 };

    String_setup_skip_chars(skip_chars, needle, nlen);

    found = String_base_search(haystack, hlen,
                 needle, nlen, skip_chars);

    return found != NULL ? found - haystack : -1;
}

StringScanner *StringScanner_create(bstring in)
{
    StringScanner *scan = calloc(1, sizeof(StringScanner));
    check_mem(scan);

    scan->in = in;
    scan->haystack = (const unsigned char *)bdata(in);
    scan->hlen = blength(in);

    assert(scan != NULL && "fuck");
    return scan;

error:
    free(scan);
    return NULL;
}

static inline void StringScanner_set_needle(StringScanner * scan,
        bstring tofind)
{
    scan->needle = (const unsigned char *)bdata(tofind);
    scan->nlen = blength(tofind);

    String_setup_skip_chars(scan->skip_chars, scan->needle, scan->nlen);
}

static inline void StringScanner_reset(StringScanner * scan)
{
    scan->haystack = (const unsigned char *)bdata(scan->in);
    scan->hlen = blength(scan->in);
}

int StringScanner_scan(StringScanner * scan, bstring tofind)
{
    const unsigned char *found = NULL;
    ssize_t found_at = 0;

    if (scan->hlen <= 0) {
        StringScanner_reset(scan);
        return -1;
    }

    if ((const unsigned char *)bdata(tofind) != scan->needle) {
        StringScanner_set_needle(scan, tofind);
    }

    found = String_base_search(scan->haystack, scan->hlen,
            scan->needle, scan->nlen,
            scan->skip_chars);

    if (found) {
        found_at = found - (const unsigned char *)bdata(scan->in);
        scan->haystack = found + scan->nlen;
        scan->hlen -= found_at - scan->nlen;
    } else {
        // done, reset the setup
        StringScanner_reset(scan);
        found_at = -1;
    }

    return found_at;
}

void StringScanner_destroy(StringScanner * scan)
{
    if (scan) {
        free(scan);
    }
}

```

.\ex39\string_algos_tests.c

```c
#include "minunit.h"
#include <lcthw/string_algos.h>
#include <lcthw/bstrlib.h>
#include <time.h>

struct tagbstring IN_STR = bsStatic(
        "I have ALPHA beta ALPHA and oranges ALPHA");
struct tagbstring ALPHA = bsStatic("ALPHA");
const int TEST_TIME = 1;

char *test_find_and_scan()
{
    StringScanner *scan = StringScanner_create(&IN_STR);
    mu_assert(scan != NULL, "Failed to make the scanner.");

    int find_i = String_find(&IN_STR, &ALPHA);
    mu_assert(find_i > 0, "Failed to find 'ALPHA' in test string.");

    int scan_i = StringScanner_scan(scan, &ALPHA);
    mu_assert(scan_i > 0, "Failed to find 'ALPHA' with scan.");
    mu_assert(scan_i == find_i, "find and scan don't match");

    scan_i = StringScanner_scan(scan, &ALPHA);
    mu_assert(scan_i > find_i,
            "should find another ALPHA after the first");

    scan_i = StringScanner_scan(scan, &ALPHA);
    mu_assert(scan_i > find_i,
            "should find another ALPHA after the first");

    mu_assert(StringScanner_scan(scan, &ALPHA) == -1,
            "shouldn't find it");

    StringScanner_destroy(scan);

    return NULL;
}

char *test_binstr_performance()
{
    int i = 0;
    int found_at = 0;
    unsigned long find_count = 0;
    time_t elapsed = 0;
    time_t start = time(NULL);

    do {
        for (i = 0; i < 1000; i++) {
            found_at = binstr(&IN_STR, 0, &ALPHA);
            mu_assert(found_at != BSTR_ERR, "Failed to find!");
            find_count++;
        }

        elapsed = time(NULL) - start;
    } while (elapsed <= TEST_TIME);

    debug("BINSTR COUNT: %lu, END TIME: %d, OPS: %f",
            find_count, (int)elapsed, (double)find_count / elapsed);
    return NULL;
}

char *test_find_performance()
{
    int i = 0;
    int found_at = 0;
    unsigned long find_count = 0;
    time_t elapsed = 0;
    time_t start = time(NULL);

    do {
        for (i = 0; i < 1000; i++) {
            found_at = String_find(&IN_STR, &ALPHA);
            find_count++;
        }

        elapsed = time(NULL) - start;
    } while (elapsed <= TEST_TIME);

    debug("FIND COUNT: %lu, END TIME: %d, OPS: %f",
            find_count, (int)elapsed, (double)find_count / elapsed);

    return NULL;
}

char *test_scan_performance()
{
    int i = 0;
    int found_at = 0;
    unsigned long find_count = 0;
    time_t elapsed = 0;
    StringScanner *scan = StringScanner_create(&IN_STR);

    time_t start = time(NULL);

    do {
        for (i = 0; i < 1000; i++) {
            found_at = 0;

            do {
                found_at = StringScanner_scan(scan, &ALPHA);
                find_count++;
            } while (found_at != -1);
        }

        elapsed = time(NULL) - start;
    } while (elapsed <= TEST_TIME);

    debug("SCAN COUNT: %lu, END TIME: %d, OPS: %f",
            find_count, (int)elapsed, (double)find_count / elapsed);

    StringScanner_destroy(scan);

    return NULL;
}

char *all_tests()
{
    mu_suite_start();

    mu_run_test(test_find_and_scan);

    // this is an idiom for commenting out sections of code
#if 0
    mu_run_test(test_scan_performance);
    mu_run_test(test_find_performance);
    mu_run_test(test_binstr_performance);
#endif

    return NULL;
}

RUN_TESTS(all_tests);

```

The code is easy to implement so should be no problem for you
at this point.  Focus on getting the unit test right.

Code Review Process

1. Start at the entry point for a piece of code that has changed.
2. For each function, confirm that its calling parameters are correct.
3. Enter that function, and confirm each line's correctness.
4. When you encounter a function, repeat up to #2 until you go no further.
5. As you exit functions, confirm the return values and their usage.
6. Continue until you are back where you started at the entry point.
7. Do a diff on your changes, and confirm any missed calls to changed functions.

Code Review Key Points

1. Check your pointer dereferences and defend against NULL.
2. Check if-statements and while loops for exiting.
3. Check return values are going to be valid.
4. Check that memory allocated is freed and other resources freed.
5. Confirm all system call parameters are correct with man pages.

Record Your Code Review

I want *you* to try to record yourself coding and reviewing your
code.  What do you learn from this experience?

What if you kept track of the number of mistakes you found in
your code reviews and analyzed the data?

Extra Credit

* See if you can make the ``Scan_find`` faster.  Why is my implementation
  here slow?
* Try some different scan times and see if you get different numbers.
  What impact does the length of time that you run the test have on
  the ``scan`` times?  What can you say about that result?
* Alter the unit test so that it runs each function for a short burst
  in the beginning to clear out any warm up period, and then start the
  timing portion.  Does that change the dependence on the length of time
  the test runs?  Does it change how many operations per second are possible?

Extra Credit

* Make the unit test randomize the strings to find and then measure
  the performance you get.  One way to do this is to use the ``bsplit``
  function from ``bstrlib.h`` to split the ``IN_STR`` on
  spaces.  Then, you can use the ``bstrList`` struct that you get to access each
  string it returns.  This will also teach you how to use ``bstrList``
  operations for string processing.
* Try some runs with the tests in different orders to see if you get different
  results.

### Exercise 40 Binary Search Trees

The Plan

Implement a Binary Search Tree, a competitor to the Hashmap.

Binary Search Trees Visually

The Code

.\ex40\bstree.h

```c
#ifndef _lcthw_BSTree_h
#define _lcthw_BSTree_h

typedef int (*BSTree_compare) (void *a, void *b);

typedef struct BSTreeNode {
    void *key;
    void *data;

    struct BSTreeNode *left;
    struct BSTreeNode *right;
    struct BSTreeNode *parent;
} BSTreeNode;

typedef struct BSTree {
    int count;
    BSTree_compare compare;
    BSTreeNode *root;
} BSTree;

typedef int (*BSTree_traverse_cb) (BSTreeNode * node);

BSTree *BSTree_create(BSTree_compare compare);
void BSTree_destroy(BSTree * map);

int BSTree_set(BSTree * map, void *key, void *data);
void *BSTree_get(BSTree * map, void *key);

int BSTree_traverse(BSTree * map, BSTree_traverse_cb traverse_cb);

void *BSTree_delete(BSTree * map, void *key);

#endif

```

.\ex40\bstree.c

```c
#include <lcthw/dbg.h>
#include <lcthw/bstree.h>
#include <stdlib.h>
#include <lcthw/bstrlib.h>

static int default_compare(void *a, void *b)
{
    return bstrcmp((bstring) a, (bstring) b);
}

BSTree *BSTree_create(BSTree_compare compare)
{
    BSTree *map = calloc(1, sizeof(BSTree));
    check_mem(map);

    map->compare = compare == NULL ? default_compare : compare;

    return map;

error:
    if (map) {
        BSTree_destroy(map);
    }
    return NULL;
}

static int BSTree_destroy_cb(BSTreeNode * node)
{
    free(node);
    return 0;
}

void BSTree_destroy(BSTree * map)
{
    if (map) {
        BSTree_traverse(map, BSTree_destroy_cb);
        free(map);
    }
}

static inline BSTreeNode *BSTreeNode_create(BSTreeNode * parent,
        void *key, void *data)
{
    BSTreeNode *node = calloc(1, sizeof(BSTreeNode));
    check_mem(node);

    node->key = key;
    node->data = data;
    node->parent = parent;
    return node;

error:
    return NULL;
}

static inline void BSTree_setnode(BSTree * map, BSTreeNode * node,
        void *key, void *data)
{
    int cmp = map->compare(node->key, key);

    if (cmp <= 0) {
        if (node->left) {
            BSTree_setnode(map, node->left, key, data);
        } else {
            node->left = BSTreeNode_create(node, key, data);
        }
    } else {
        if (node->right) {
            BSTree_setnode(map, node->right, key, data);
        } else {
            node->right = BSTreeNode_create(node, key, data);
        }
    }
}

int BSTree_set(BSTree * map, void *key, void *data)
{
    if (map->root == NULL) {
        // first so just make it and get out
        map->root = BSTreeNode_create(NULL, key, data);
        check_mem(map->root);
    } else {
        BSTree_setnode(map, map->root, key, data);
    }

    return 0;
error:
    return -1;
}

static inline BSTreeNode *BSTree_getnode(BSTree * map,
        BSTreeNode * node, void *key)
{
    int cmp = map->compare(node->key, key);

    if (cmp == 0) {
        return node;
    } else if (cmp < 0) {
        if (node->left) {
            return BSTree_getnode(map, node->left, key);
        } else {
            return NULL;
        }
    } else {
        if (node->right) {
            return BSTree_getnode(map, node->right, key);
        } else {
            return NULL;
        }
    }
}

void *BSTree_get(BSTree * map, void *key)
{
    if (map->root == NULL) {
        return NULL;
    } else {
        BSTreeNode *node = BSTree_getnode(map, map->root, key);
        return node == NULL ? NULL : node->data;
    }
}

static inline int BSTree_traverse_nodes(BSTreeNode * node,
        BSTree_traverse_cb traverse_cb)
{
    int rc = 0;

    if (node->left) {
        rc = BSTree_traverse_nodes(node->left, traverse_cb);
        if (rc != 0)
            return rc;
    }

    if (node->right) {
        rc = BSTree_traverse_nodes(node->right, traverse_cb);
        if (rc != 0)
            return rc;
    }

    return traverse_cb(node);
}

int BSTree_traverse(BSTree * map, BSTree_traverse_cb traverse_cb)
{
    if (map->root) {
        return BSTree_traverse_nodes(map->root, traverse_cb);
    }

    return 0;
}

static inline BSTreeNode *BSTree_find_min(BSTreeNode * node)
{
    while (node->left) {
        node = node->left;
    }

    return node;
}

static inline void BSTree_replace_node_in_parent(BSTree * map,
        BSTreeNode * node,
        BSTreeNode * new_value)
{
    if (node->parent) {
        if (node == node->parent->left) {
            node->parent->left = new_value;
        } else {
            node->parent->right = new_value;
        }
    } else {
        // this is the root so gotta change it
        map->root = new_value;
    }

    if (new_value) {
        new_value->parent = node->parent;
    }
}

static inline void BSTree_swap(BSTreeNode * a, BSTreeNode * b)
{
    void *temp = NULL;
    temp = b->key;
    b->key = a->key;
    a->key = temp;
    temp = b->data;
    b->data = a->data;
    a->data = temp;
}

static inline BSTreeNode *BSTree_node_delete(BSTree * map,
        BSTreeNode * node,
        void *key)
{
    int cmp = map->compare(node->key, key);

    if (cmp < 0) {
        if (node->left) {
            return BSTree_node_delete(map, node->left, key);
        } else {
            // not found
            return NULL;
        }
    } else if (cmp > 0) {
        if (node->right) {
            return BSTree_node_delete(map, node->right, key);
        } else {
            // not found
            return NULL;
        }
    } else {
        if (node->left && node->right) {
            // swap this node for the smallest node that is bigger than us
            BSTreeNode *successor = BSTree_find_min(node->right);
            BSTree_swap(successor, node);

            // this leaves the old successor with possibly a right child
            // so replace it with that right child
            BSTree_replace_node_in_parent(map, successor,
                    successor->right);

            // finally it's swapped, so return successor instead of node
            return successor;
        } else if (node->left) {
            BSTree_replace_node_in_parent(map, node, node->left);
        } else if (node->right) {
            BSTree_replace_node_in_parent(map, node, node->right);
        } else {
            BSTree_replace_node_in_parent(map, node, NULL);
        }

        return node;
    }
}

void *BSTree_delete(BSTree * map, void *key)
{
    void *data = NULL;

    if (map->root) {
        BSTreeNode *node = BSTree_node_delete(map, map->root, key);

        if (node) {
            data = node->data;
            free(node);
        }
    }

    return data;
}

```

.\ex40\bstree_tests.c

```c
#include "minunit.h"
#include <lcthw/bstree.h>
#include <assert.h>
#include <lcthw/bstrlib.h>
#include <stdlib.h>
#include <time.h>

BSTree *map = NULL;
static int traverse_called = 0;
struct tagbstring test1 = bsStatic("test data 1");
struct tagbstring test2 = bsStatic("test data 2");
struct tagbstring test3 = bsStatic("xest data 3");
struct tagbstring expect1 = bsStatic("THE VALUE 1");
struct tagbstring expect2 = bsStatic("THE VALUE 2");
struct tagbstring expect3 = bsStatic("THE VALUE 3");

static int traverse_good_cb(BSTreeNode * node)
{
    debug("KEY: %s", bdata((bstring) node->key));
    traverse_called++;
    return 0;
}

static int traverse_fail_cb(BSTreeNode * node)
{
    debug("KEY: %s", bdata((bstring) node->key));
    traverse_called++;

    if (traverse_called == 2) {
        return 1;
    } else {
        return 0;
    }
}

char *test_create()
{
    map = BSTree_create(NULL);
    mu_assert(map != NULL, "Failed to create map.");

    return NULL;
}

char *test_destroy()
{
    BSTree_destroy(map);

    return NULL;
}

char *test_get_set()
{
    int rc = BSTree_set(map, &test1, &expect1);
    mu_assert(rc == 0, "Failed to set &test1");
    bstring result = BSTree_get(map, &test1);
    mu_assert(result == &expect1, "Wrong value for test1.");

    rc = BSTree_set(map, &test2, &expect2);
    mu_assert(rc == 0, "Failed to set test2");
    result = BSTree_get(map, &test2);
    mu_assert(result == &expect2, "Wrong value for test2.");

    rc = BSTree_set(map, &test3, &expect3);
    mu_assert(rc == 0, "Failed to set test3");
    result = BSTree_get(map, &test3);
    mu_assert(result == &expect3, "Wrong value for test3.");

    return NULL;
}

char *test_traverse()
{
    int rc = BSTree_traverse(map, traverse_good_cb);
    mu_assert(rc == 0, "Failed to traverse.");
    mu_assert(traverse_called == 3, "Wrong count traverse.");

    traverse_called = 0;
    rc = BSTree_traverse(map, traverse_fail_cb);
    mu_assert(rc == 1, "Failed to traverse.");
    mu_assert(traverse_called == 2, "Wrong count traverse for fail.");

    return NULL;
}

char *test_delete()
{
    bstring deleted = (bstring) BSTree_delete(map, &test1);
    mu_assert(deleted != NULL, "Got NULL on delete.");
    mu_assert(deleted == &expect1, "Should get test1");
    bstring result = BSTree_get(map, &test1);
    mu_assert(result == NULL, "Should delete.");

    deleted = (bstring) BSTree_delete(map, &test1);
    mu_assert(deleted == NULL, "Should get NULL on delete");

    deleted = (bstring) BSTree_delete(map, &test2);
    mu_assert(deleted != NULL, "Got NULL on delete.");
    mu_assert(deleted == &expect2, "Should get test2");
    result = BSTree_get(map, &test2);
    mu_assert(result == NULL, "Should delete.");

    deleted = (bstring) BSTree_delete(map, &test3);
    mu_assert(deleted != NULL, "Got NULL on delete.");
    mu_assert(deleted == &expect3, "Should get test3");
    result = BSTree_get(map, &test3);
    mu_assert(result == NULL, "Should delete.");

    // test deleting non-existent stuff
    deleted = (bstring) BSTree_delete(map, &test3);
    mu_assert(deleted == NULL, "Should get NULL");

    return NULL;
}

char *test_fuzzing()
{
    BSTree *store = BSTree_create(NULL);
    int i = 0;
    int j = 0;
    bstring numbers[100] = { NULL };
    bstring data[100] = { NULL };
    srand((unsigned int)time(NULL));

    for (i = 0; i < 100; i++) {
        int num = rand();
        numbers[i] = bformat("%d", num);
        data[i] = bformat("data %d", num);
        BSTree_set(store, numbers[i], data[i]);
    }

    for (i = 0; i < 100; i++) {
        bstring value = BSTree_delete(store, numbers[i]);
        mu_assert(value == data[i],
                "Failed to delete the right number.");

        mu_assert(BSTree_delete(store, numbers[i]) == NULL,
                "Should get nothing.");

        for (j = i + 1; j < 99 - i; j++) {
            bstring value = BSTree_get(store, numbers[j]);
            mu_assert(value == data[j],
                    "Failed to get the right number.");
        }

        bdestroy(value);
        bdestroy(numbers[i]);
    }

    BSTree_destroy(store);

    return NULL;
}

char *all_tests()
{
    mu_suite_start();

    mu_run_test(test_create);
    mu_run_test(test_get_set);
    mu_run_test(test_traverse);
    mu_run_test(test_delete);
    mu_run_test(test_destroy);
    mu_run_test(test_fuzzing);

    return NULL;
}

RUN_TESTS(all_tests);

```

There's nothing new in the code, but make sure you read the book carefully.

Code Review

.\ex40\bstree.h

```c
#ifndef _lcthw_BSTree_h
#define _lcthw_BSTree_h

typedef int (*BSTree_compare) (void *a, void *b);

typedef struct BSTreeNode {
    void *key;
    void *data;

    struct BSTreeNode *left;
    struct BSTreeNode *right;
    struct BSTreeNode *parent;
} BSTreeNode;

typedef struct BSTree {
    int count;
    BSTree_compare compare;
    BSTreeNode *root;
} BSTree;

typedef int (*BSTree_traverse_cb) (BSTreeNode * node);

BSTree *BSTree_create(BSTree_compare compare);
void BSTree_destroy(BSTree * map);

int BSTree_set(BSTree * map, void *key, void *data);
void *BSTree_get(BSTree * map, void *key);

int BSTree_traverse(BSTree * map, BSTree_traverse_cb traverse_cb);

void *BSTree_delete(BSTree * map, void *key);

#endif

```

.\ex40\bstree.c

```c
#include <lcthw/dbg.h>
#include <lcthw/bstree.h>
#include <stdlib.h>
#include <lcthw/bstrlib.h>

static int default_compare(void *a, void *b)
{
    return bstrcmp((bstring) a, (bstring) b);
}

BSTree *BSTree_create(BSTree_compare compare)
{
    BSTree *map = calloc(1, sizeof(BSTree));
    check_mem(map);

    map->compare = compare == NULL ? default_compare : compare;

    return map;

error:
    if (map) {
        BSTree_destroy(map);
    }
    return NULL;
}

static int BSTree_destroy_cb(BSTreeNode * node)
{
    free(node);
    return 0;
}

void BSTree_destroy(BSTree * map)
{
    if (map) {
        BSTree_traverse(map, BSTree_destroy_cb);
        free(map);
    }
}

static inline BSTreeNode *BSTreeNode_create(BSTreeNode * parent,
        void *key, void *data)
{
    BSTreeNode *node = calloc(1, sizeof(BSTreeNode));
    check_mem(node);

    node->key = key;
    node->data = data;
    node->parent = parent;
    return node;

error:
    return NULL;
}

static inline void BSTree_setnode(BSTree * map, BSTreeNode * node,
        void *key, void *data)
{
    int cmp = map->compare(node->key, key);

    if (cmp <= 0) {
        if (node->left) {
            BSTree_setnode(map, node->left, key, data);
        } else {
            node->left = BSTreeNode_create(node, key, data);
        }
    } else {
        if (node->right) {
            BSTree_setnode(map, node->right, key, data);
        } else {
            node->right = BSTreeNode_create(node, key, data);
        }
    }
}

int BSTree_set(BSTree * map, void *key, void *data)
{
    if (map->root == NULL) {
        // first so just make it and get out
        map->root = BSTreeNode_create(NULL, key, data);
        check_mem(map->root);
    } else {
        BSTree_setnode(map, map->root, key, data);
    }

    return 0;
error:
    return -1;
}

static inline BSTreeNode *BSTree_getnode(BSTree * map,
        BSTreeNode * node, void *key)
{
    int cmp = map->compare(node->key, key);

    if (cmp == 0) {
        return node;
    } else if (cmp < 0) {
        if (node->left) {
            return BSTree_getnode(map, node->left, key);
        } else {
            return NULL;
        }
    } else {
        if (node->right) {
            return BSTree_getnode(map, node->right, key);
        } else {
            return NULL;
        }
    }
}

void *BSTree_get(BSTree * map, void *key)
{
    if (map->root == NULL) {
        return NULL;
    } else {
        BSTreeNode *node = BSTree_getnode(map, map->root, key);
        return node == NULL ? NULL : node->data;
    }
}

static inline int BSTree_traverse_nodes(BSTreeNode * node,
        BSTree_traverse_cb traverse_cb)
{
    int rc = 0;

    if (node->left) {
        rc = BSTree_traverse_nodes(node->left, traverse_cb);
        if (rc != 0)
            return rc;
    }

    if (node->right) {
        rc = BSTree_traverse_nodes(node->right, traverse_cb);
        if (rc != 0)
            return rc;
    }

    return traverse_cb(node);
}

int BSTree_traverse(BSTree * map, BSTree_traverse_cb traverse_cb)
{
    if (map->root) {
        return BSTree_traverse_nodes(map->root, traverse_cb);
    }

    return 0;
}

static inline BSTreeNode *BSTree_find_min(BSTreeNode * node)
{
    while (node->left) {
        node = node->left;
    }

    return node;
}

static inline void BSTree_replace_node_in_parent(BSTree * map,
        BSTreeNode * node,
        BSTreeNode * new_value)
{
    if (node->parent) {
        if (node == node->parent->left) {
            node->parent->left = new_value;
        } else {
            node->parent->right = new_value;
        }
    } else {
        // this is the root so gotta change it
        map->root = new_value;
    }

    if (new_value) {
        new_value->parent = node->parent;
    }
}

static inline void BSTree_swap(BSTreeNode * a, BSTreeNode * b)
{
    void *temp = NULL;
    temp = b->key;
    b->key = a->key;
    a->key = temp;
    temp = b->data;
    b->data = a->data;
    a->data = temp;
}

static inline BSTreeNode *BSTree_node_delete(BSTree * map,
        BSTreeNode * node,
        void *key)
{
    int cmp = map->compare(node->key, key);

    if (cmp < 0) {
        if (node->left) {
            return BSTree_node_delete(map, node->left, key);
        } else {
            // not found
            return NULL;
        }
    } else if (cmp > 0) {
        if (node->right) {
            return BSTree_node_delete(map, node->right, key);
        } else {
            // not found
            return NULL;
        }
    } else {
        if (node->left && node->right) {
            // swap this node for the smallest node that is bigger than us
            BSTreeNode *successor = BSTree_find_min(node->right);
            BSTree_swap(successor, node);

            // this leaves the old successor with possibly a right child
            // so replace it with that right child
            BSTree_replace_node_in_parent(map, successor,
                    successor->right);

            // finally it's swapped, so return successor instead of node
            return successor;
        } else if (node->left) {
            BSTree_replace_node_in_parent(map, node, node->left);
        } else if (node->right) {
            BSTree_replace_node_in_parent(map, node, node->right);
        } else {
            BSTree_replace_node_in_parent(map, node, NULL);
        }

        return node;
    }
}

void *BSTree_delete(BSTree * map, void *key)
{
    void *data = NULL;

    if (map->root) {
        BSTreeNode *node = BSTree_node_delete(map, map->root, key);

        if (node) {
            data = node->data;
            free(node);
        }
    }

    return data;
}

```

.\ex40\bstree_tests.c

```c
#include "minunit.h"
#include <lcthw/bstree.h>
#include <assert.h>
#include <lcthw/bstrlib.h>
#include <stdlib.h>
#include <time.h>

BSTree *map = NULL;
static int traverse_called = 0;
struct tagbstring test1 = bsStatic("test data 1");
struct tagbstring test2 = bsStatic("test data 2");
struct tagbstring test3 = bsStatic("xest data 3");
struct tagbstring expect1 = bsStatic("THE VALUE 1");
struct tagbstring expect2 = bsStatic("THE VALUE 2");
struct tagbstring expect3 = bsStatic("THE VALUE 3");

static int traverse_good_cb(BSTreeNode * node)
{
    debug("KEY: %s", bdata((bstring) node->key));
    traverse_called++;
    return 0;
}

static int traverse_fail_cb(BSTreeNode * node)
{
    debug("KEY: %s", bdata((bstring) node->key));
    traverse_called++;

    if (traverse_called == 2) {
        return 1;
    } else {
        return 0;
    }
}

char *test_create()
{
    map = BSTree_create(NULL);
    mu_assert(map != NULL, "Failed to create map.");

    return NULL;
}

char *test_destroy()
{
    BSTree_destroy(map);

    return NULL;
}

char *test_get_set()
{
    int rc = BSTree_set(map, &test1, &expect1);
    mu_assert(rc == 0, "Failed to set &test1");
    bstring result = BSTree_get(map, &test1);
    mu_assert(result == &expect1, "Wrong value for test1.");

    rc = BSTree_set(map, &test2, &expect2);
    mu_assert(rc == 0, "Failed to set test2");
    result = BSTree_get(map, &test2);
    mu_assert(result == &expect2, "Wrong value for test2.");

    rc = BSTree_set(map, &test3, &expect3);
    mu_assert(rc == 0, "Failed to set test3");
    result = BSTree_get(map, &test3);
    mu_assert(result == &expect3, "Wrong value for test3.");

    return NULL;
}

char *test_traverse()
{
    int rc = BSTree_traverse(map, traverse_good_cb);
    mu_assert(rc == 0, "Failed to traverse.");
    mu_assert(traverse_called == 3, "Wrong count traverse.");

    traverse_called = 0;
    rc = BSTree_traverse(map, traverse_fail_cb);
    mu_assert(rc == 1, "Failed to traverse.");
    mu_assert(traverse_called == 2, "Wrong count traverse for fail.");

    return NULL;
}

char *test_delete()
{
    bstring deleted = (bstring) BSTree_delete(map, &test1);
    mu_assert(deleted != NULL, "Got NULL on delete.");
    mu_assert(deleted == &expect1, "Should get test1");
    bstring result = BSTree_get(map, &test1);
    mu_assert(result == NULL, "Should delete.");

    deleted = (bstring) BSTree_delete(map, &test1);
    mu_assert(deleted == NULL, "Should get NULL on delete");

    deleted = (bstring) BSTree_delete(map, &test2);
    mu_assert(deleted != NULL, "Got NULL on delete.");
    mu_assert(deleted == &expect2, "Should get test2");
    result = BSTree_get(map, &test2);
    mu_assert(result == NULL, "Should delete.");

    deleted = (bstring) BSTree_delete(map, &test3);
    mu_assert(deleted != NULL, "Got NULL on delete.");
    mu_assert(deleted == &expect3, "Should get test3");
    result = BSTree_get(map, &test3);
    mu_assert(result == NULL, "Should delete.");

    // test deleting non-existent stuff
    deleted = (bstring) BSTree_delete(map, &test3);
    mu_assert(deleted == NULL, "Should get NULL");

    return NULL;
}

char *test_fuzzing()
{
    BSTree *store = BSTree_create(NULL);
    int i = 0;
    int j = 0;
    bstring numbers[100] = { NULL };
    bstring data[100] = { NULL };
    srand((unsigned int)time(NULL));

    for (i = 0; i < 100; i++) {
        int num = rand();
        numbers[i] = bformat("%d", num);
        data[i] = bformat("data %d", num);
        BSTree_set(store, numbers[i], data[i]);
    }

    for (i = 0; i < 100; i++) {
        bstring value = BSTree_delete(store, numbers[i]);
        mu_assert(value == data[i],
                "Failed to delete the right number.");

        mu_assert(BSTree_delete(store, numbers[i]) == NULL,
                "Should get nothing.");

        for (j = i + 1; j < 99 - i; j++) {
            bstring value = BSTree_get(store, numbers[j]);
            mu_assert(value == data[j],
                    "Failed to get the right number.");
        }

        bdestroy(value);
        bdestroy(numbers[i]);
    }

    BSTree_destroy(store);

    return NULL;
}

char *all_tests()
{
    mu_suite_start();

    mu_run_test(test_create);
    mu_run_test(test_get_set);
    mu_run_test(test_traverse);
    mu_run_test(test_delete);
    mu_run_test(test_destroy);
    mu_run_test(test_fuzzing);

    return NULL;
}

RUN_TESTS(all_tests);

```

I'll walk through the implementation and compare it to Hashmaps for features.

Improving It

* As usual, you should go through all of the defensive programming checks and add
  *assert*s for conditions that shouldn't happen.  For example, you shouldn't be getting *NULL* values for the recursion functions, so assert that.
* The traverse function walks through the tree in order by traversing left, then right,
  and then the current node.  You can create traverse functions for the reverse order, as well.

Improving It

* It does a full string compare on every node, but I could use the *Hashmap*
  hashing functions to speed this up.  I could hash the keys, and then keep the hash in
  the *BSTreeNode*.  Then, in each of the set up functions, I can hash the
  key ahead of time, and pass it down to the recursive function.  Using this hash, I can
  then compare each node much quicker in a way that's similar to what I do in *Hashmap*.

Breaking It

A big flaw in this is the use of recursion.  An attacker could choose data to cause a stack overflow.

Extra Credit

* There's an alternative way to do this data structure without using recursion.  The Wikipedia
  page shows alternatives that don't use recursion but do the same thing.  Why would this
  be better or worse?
* Read up on all of the different but similar trees you can find. There are AVL trees (named after <NAME> and <NAME>), red-black trees,
  and some non-tree structures like skip lists.

<file_sep>+++
title = "Coding"
weight = 2
+++

{{%children style="card" description="true" %}}<file_sep>#!/bin/bash
#******************************************************************************
#    AWS VPC Creation Shell Script
#******************************************************************************
#
# SYNOPSIS
#    Automates the creation of a custom IPv4 VPC, having both a public and a
#    private subnet, and a NAT gateway.
#
#==============================================================================
#
# NOTES
#   VERSION:   1.0
#   AUTHOR:    <NAME>
#
#==============================================================================
#   MODIFY THE SETTINGS BELOW
#==============================================================================
#
AWS_REGION="ap-southeast-2"
VPC_NAME="DEV-PG-II"
VPC_CIDR="10.5.0.0/16"
SUBNET_PUBLIC_CIDR="10.5.1.0/24"
SUBNET_PUBLIC_AZ="ap-southeast-2a"
SUBNET_PUBLIC_NAME="$VPC_NAME-PubSub-AZ2a"
SUBNET_PRIVATE_CIDR="10.5.2.0/24"
SUBNET_PRIVATE_AZ="ap-southeast-2b"
SUBNET_PRIVATE_NAME="$VPC_NAME-PrvSub-AZ2b"
IGW_NAME="$VPC_NAME-IGW"
NAT_GW_NAME="$VPC_NAME-NAT-GW"
CHECK_FREQUENCY=5
#
#==============================================================================
#   DO NOT MODIFY CODE BELOW
#==============================================================================
#
# Create VPC
echo "Creating VPC in preferred region..."
VPC_ID=$(aws ec2 create-vpc \
  --cidr-block $VPC_CIDR \
  --query 'Vpc.{VpcId:VpcId}' \
  --output text \
  --region $AWS_REGION)
echo "  VPC ID '$VPC_ID' CREATED in '$AWS_REGION' region."

# Add Name tag to VPC
aws ec2 create-tags \
  --resources $VPC_ID \
  --tags "Key=Name,Value=$VPC_NAME" \
  --region $AWS_REGION
echo "  VPC ID '$VPC_ID' NAMED as '$VPC_NAME'."

# Create Public Subnet
echo "Creating Public Subnet..."
SUBNET_PUBLIC_ID=$(aws ec2 create-subnet \
  --vpc-id $VPC_ID \
  --cidr-block $SUBNET_PUBLIC_CIDR \
  --availability-zone $SUBNET_PUBLIC_AZ \
  --query 'Subnet.{SubnetId:SubnetId}' \
  --output text \
  --region $AWS_REGION)
echo "  Subnet ID '$SUBNET_PUBLIC_ID' CREATED in '$SUBNET_PUBLIC_AZ'" \
  "Availability Zone."

# Add Name tag to Public Subnet
aws ec2 create-tags \
  --resources $SUBNET_PUBLIC_ID \
  --tags "Key=Name,Value=$SUBNET_PUBLIC_NAME" \
  --region $AWS_REGION
echo "  Subnet ID '$SUBNET_PUBLIC_ID' NAMED as" \
  "'$SUBNET_PUBLIC_NAME'."

# Create Private Subnet
echo "Creating Private Subnet..."
SUBNET_PRIVATE_ID=$(aws ec2 create-subnet \
  --vpc-id $VPC_ID \
  --cidr-block $SUBNET_PRIVATE_CIDR \
  --availability-zone $SUBNET_PRIVATE_AZ \
  --query 'Subnet.{SubnetId:SubnetId}' \
  --output text \
  --region $AWS_REGION)
echo "  Subnet ID '$SUBNET_PRIVATE_ID' CREATED in '$SUBNET_PRIVATE_AZ'" \
  "Availability Zone."

# Add Name tag to Private Subnet
aws ec2 create-tags \
  --resources $SUBNET_PRIVATE_ID \
  --tags "Key=Name,Value=$SUBNET_PRIVATE_NAME" \
  --region $AWS_REGION
echo "  Subnet ID '$SUBNET_PRIVATE_ID' NAMED as '$SUBNET_PRIVATE_NAME'."

# Create Internet gateway
echo "Creating Internet Gateway..."
IGW_ID=$(aws ec2 create-internet-gateway \
  --query 'InternetGateway.{InternetGatewayId:InternetGatewayId}' \
  --output text \
  --region $AWS_REGION)
echo "  Internet Gateway ID '$IGW_ID' CREATED."

# Add Name tag to Internet gateway
aws ec2 create-tags \
  --resources $IGW_ID \
  --tags "Key=Name,Value=$IGW_NAME" \
  --region $AWS_REGION
echo " Internet gateway '$IGW_ID' NAMED as '$IGW_NAME'."


# Attach Internet gateway to your VPC
aws ec2 attach-internet-gateway \
  --vpc-id $VPC_ID \
  --internet-gateway-id $IGW_ID \
  --region $AWS_REGION
echo "  Internet Gateway ID '$IGW_ID' ATTACHED to VPC ID '$VPC_ID'."

# Create Route Table
echo "Creating Route Table..."
ROUTE_TABLE_ID=$(aws ec2 create-route-table \
  --vpc-id $VPC_ID \
  --query 'RouteTable.{RouteTableId:RouteTableId}' \
  --output text \
  --region $AWS_REGION)
echo "  Route Table ID '$ROUTE_TABLE_ID' CREATED."


# Create route to Internet Gateway
RESULT=$(aws ec2 create-route \
  --route-table-id $ROUTE_TABLE_ID \
  --destination-cidr-block 0.0.0.0/0 \
  --gateway-id $IGW_ID \
  --region $AWS_REGION)
echo "  Route to '0.0.0.0/0' via Internet Gateway ID '$IGW_ID' ADDED to" \
  "Route Table ID '$ROUTE_TABLE_ID'."

# Associate Public Subnet with Route Table
RESULT=$(aws ec2 associate-route-table  \
  --subnet-id $SUBNET_PUBLIC_ID \
  --route-table-id $ROUTE_TABLE_ID \
  --region $AWS_REGION)
echo "  Public Subnet ID '$SUBNET_PUBLIC_ID' ASSOCIATED with Route Table ID" \
  "'$ROUTE_TABLE_ID'."

# Enable Auto-assign Public IP on Public Subnet
aws ec2 modify-subnet-attribute \
  --subnet-id $SUBNET_PUBLIC_ID \
  --map-public-ip-on-launch \
  --region $AWS_REGION
echo "  'Auto-assign Public IP' ENABLED on Public Subnet ID" \
  "'$SUBNET_PUBLIC_ID'."

# Allocate Elastic IP Address for NAT Gateway
echo "Creating NAT Gateway..."
EIP_ALLOC_ID=$(aws ec2 allocate-address \
  --domain vpc \
  --query '{AllocationId:AllocationId}' \
  --output text \
  --region $AWS_REGION)
echo "  Elastic IP address ID '$EIP_ALLOC_ID' ALLOCATED."

# Create NAT Gateway
NAT_GW_ID=$(aws ec2 create-nat-gateway \
  --subnet-id $SUBNET_PUBLIC_ID \
  --allocation-id $EIP_ALLOC_ID \
  --query 'NatGateway.{NatGatewayId:NatGatewayId}' \
  --output text \
  --region $AWS_REGION)

FORMATTED_MSG="Creating NAT Gateway ID '$NAT_GW_ID' and waiting for it to "
FORMATTED_MSG+="become available.\n    Please BE PATIENT as this can take some "
FORMATTED_MSG+="time to complete.\n    ......\n"
printf "  $FORMATTED_MSG"
FORMATTED_MSG="STATUS: AVAILABLE - Total of %02d seconds elapsed for process"
FORMATTED_MSG+="\n    ......\n  NAT Gateway ID '%s' is now AVAILABLE.\n"
start_time="$(date -u +%s)"

aws ec2 wait nat-gateway-available \
  --nat-gateway-ids $NAT_GW_ID

end_time="$(date -u +%s)"
elapsed="$(($end_time-$start_time))" 

printf "  $FORMATTED_MSG" $elapsed $NAT_GW_ID

# Add Name tag to NAT Gateway
aws ec2 create-tags \
  --resources $NAT_GW_ID \
  --tags "Key=Name,Value=$NAT_GW_NAME" \
  --region $AWS_REGION
echo " Internet gateway '$NAT_GW_ID' NAMED as '$NAT_GW_NAME'."


# Create route to NAT Gateway
MAIN_ROUTE_TABLE_ID=$(aws ec2 describe-route-tables \
  --filters Name=vpc-id,Values=$VPC_ID Name=association.main,Values=true \
  --query 'RouteTables[*].{RouteTableId:RouteTableId}' \
  --output text \
  --region $AWS_REGION)
echo "  Main Route Table ID is '$MAIN_ROUTE_TABLE_ID'."
RESULT=$(aws ec2 create-route \
  --route-table-id $MAIN_ROUTE_TABLE_ID \
  --destination-cidr-block 0.0.0.0/0 \
  --gateway-id $NAT_GW_ID \
  --region $AWS_REGION)
echo "  Route to '0.0.0.0/0' via NAT Gateway with ID '$NAT_GW_ID' ADDED to" \
  "Route Table ID '$MAIN_ROUTE_TABLE_ID'."
echo "COMPLETED"

<file_sep>+++
date = "2016-03-04T14:59:31+11:00"
title = "Ubuntu 14 -- server setup"
description = "Ubuntu 14 -- server  note"
draft = false
+++

Prelude

> *This article is mainly to help experienced user install and setup Ubuntu server. If you are not familiar with Ubuntu system, please install Ubuntu desktop version at first, and you can follow [Ubuntu deskstop setup](/os/ubuntu-desktop-14/)*

## Prerequisites

* You are familiar with Ubuntu, at least you have some experience working on Linux system. 
* You are familiar with bash/shell script
* You are going to setup Ubuntu server for special purpose. e.g. Web server, file server, or data center.  


## UFW setup

```bash
sudo ufw enable
sudo ufw allow 80/tcp
sudo ufw allow ssh
sudo ufw allow 443/tcp
sudo ufw allow 8000/tcp
```

## SSH server setup

### Secure SSH with CA in production

```bash
sudo apt-get install openssh-server 

## backup default config 
sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.factory-defaults
sudo chmod a-w /etc/ssh/sshd_config.factory-defaults

## use any editor to update sshd_config 
sudo nano /etc/ssh/sshd_config

## uncomment  PasswordAuthentication yes to allow remote password login
## Password authentication is only for test environment

## setup ssh auto-start onboot
sudo update-rc.d ssh defaults
```


## Time Zone setup

```bash
sudo dpkg-reconfigure tzdata
```

## install software-properties-common Package

```bash
software-properties-common python-software-properties
```

Install byobu screen

```bash
sudo apt-get install byobu screen 

## Launch byobu
byobu
## F9 for help 
## change the keyboard for putty > Termianl > Keyboard > Function keys and keyboard > Xterm R6
```


## install docker (Ubuntu 14.04 LTS)

```bash
## add GBG Key
sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys <KEY>

## add docker.list
sudo touch /etc/apt/sources.list.d/docker.list
##  repo 
sudo vi /etc/apt/sources.list.d/docker.list
##  add following repo ath the end of file 
deb https://apt.dockerproject.org/repo ubuntu-trusty main

## apt update 
sudo apt-get update
## verify 
apt-cache policy docker-engine
## install docker engine

```


## build vim

* I am not `vi` fans, but if you really want to use `vi`. I wiil suggest spend some time to dig into [`vimawesome`](http://vimawesome.com) and play around with those plugins. Some are pretty cool, e.g. `NERD Tree`, 'youcompleteme`, `syntastic`, etc. 

```bash
sudo apt-get build-dep vim
git clone https://github.com/vim/vim.git ~/forks/vim
cd ~/forks/vim
### make distclean && make clean
### build script from this repo
make VIMRUNTIMEDIR=/usr/share/vim/vim74
sudo make install
```

## Install JDK 8 

* If you are going to run Java Web Application on server, or you are going to setup Hadoop environment. 
* Setup oracle jdk ppa and install oracle jdk from ppa.

```bash
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer
sudo apt-get install oracle-java8-set-default     
```

## Install OpenJdk 

* Setup OpenJdk ppa and install it from ppa

```bash
sudo add-apt-repository ppa:openjdk-r/ppa
sudo apt-get update 
sudo apt-get install openjdk-8-jdk
sudo update-alternatives --config java
```
* Type in a number to select a Java version.
* set default Java Compiler 

```bash
sudo update-alternatives --config javac
java -version
```

* How to stop mysql auto startup 

* Comment out the line below in the config file ( /etc/init/mysql.conf )
   * Start on (net-device-up


## Install nodejs

```bash
curl -sL https://deb.nodesource.com/setup | sudo bash -
sudo apt-get install nodejs
sudo apt-get install build-essential
```

Setup NPM

* You can use default `npm` on your server after you install nodejs, but there is a better way to manage your `npm`. It allows you easily to control your packages.  

```bash
cd 
mkdir .node_modules
npm config list
npm config get prefix
## /usr or  /usr/share
npm config set prefix $HOME/.node_modules
cat .npmrc
## /home/hho/.node_modules
npm install -g npm
which npm
## /usr/bin/npm
```

* Open .profile add following to end of file

```bash
export PATH="$HOME/.node_modules_global/bin:$PATH"
```

## Install nvm

* Using `nvm` is no longer popular and best option. I will recommand you just use `npm` to manage eveything you need.

```bash
sudo apt-get update
sudo apt-get install build-essential libssl-dev
curl https://raw.githubusercontent.com/creationix/nvm/v0.16.1/install.sh | sh
source ~/.profile
nvm ls-remote
nvm install 0.11.13
nvm use 0.11.13
nvm alias default 0.11.13
nvm use default
```

## Install PHP & Compser 

```bash
sudo apt-get install curl php5-cli git
curl -sS https://getcomposer.org/installer | sudo php -- --install-dir=/usr/local/bin --filename=composer
```

## Install Python2, Python3

* Ubuntu has python instaleld by default

```bash
sudo apt-get python pip 
sudo apt-get install python3 pip3
## Install virtualenv 
sudo pip install virtualenv
sudo pip3 install virtualenv
```



## Install Go

```bash
wget https://storage.googleapis.com/golang/go1.4.linux-amd64.tar.gz
sudo tar -xzf go1.4.linux-amd64.tar.gz -C /usr/local

sudo vi /etc/profile
GOPATH="/YOUR/USER/HOME/go"
GOROOT="/usr/local/go"
PATH=$GOROOT/bin:$PATH
```

## Install R 

```bash
sudo apt-key adv –keyserver keyserver.ubuntu.com –recv-keys E084DAB9
sudo add-apt-repository ‘deb http://star-www.st-andrews.ac.uk/cran/bin/linux/ubuntu trusty/’
sudo apt-get update
sudo apt-get install r-base
```


## Install Rust

```bash
$ curl -sf -L https://static.rust-lang.org/rustup.sh | sh
```
*Uninstall Rust*
```bash
$ sudo /usr/local/lib/rustlib/uninstall.sh
```

<file_sep>+++
date = "2017-06-07T16:56:21+11:00"
title = "What is SRE"
description="SRE - Site Reliability Engineering"

draft = true
+++

<file_sep>+++
title = "Azure: CAF - 2"
weight = 1
description="Function & Team  of Cloud Adoption Framework"
+++


## Function

* Cloud adoption functions deliver technical solutions.
* Cloud strategy functions align technical change with business needs.
* Cloud operations functions support and operate adopted solutions.
* Cloud center of excellence (CCoE) functions improve quality, speed, and resiliency of adoption.
* Cloud governance functions manage risk.
* Cloud platform functions operate and mature the platform.
* Cloud automation functions accelerate adoption and innovation.
* Cloud security functions manage security risks.


## Team

### cloud strategy team

To be successful, every cloud adoption journey needs to involve some level of strategic planning. This getting started guide is designed to help you establish a dedicated team or virtual team that can build and deliver on a solid cloud strategy.

As the cloud strategy team forms and gets started, the strategy facilitator is temporarily accountable for creating alignment and keeping the team aligned with business goals. The strategy facilitator is often the person most accountable for the success of the cloud adoption journey.

> Note: Not every organization requires a dedicated team or virtual team to help meet its strategic needs. In your RACI (responsible, accountable, consulted, and informed) template, list the core accountabilities of the strategy, and identify the person on your team who will be accountable for each. If one person will take on all of those accountabilities, simple replace "cloud strategy" with that person's name in the RACI template.

* Step 1: Determine whether a cloud strategy team is needed

    Deliverables: Determine whether your business requires a cloud strategy team

* Step 2: Establish the cloud strategy team

    Deliverables: Identify the appropriate organizations or individuals who are willing to share in the accountability and responsibility for driving the cloud adoption strategy.

* Step 3: Establish a cadence

    Deliverables: Review suggested meeting cadences and schedule meetings with all strategy team participants.

* Step 4: Establish a motivation-driven strategy

    Deliverables: Record motivations in the strategy and plan template.

* Step 5: Establish business outcomes

    Deliverables:
    - Identify at least one expected business outcome per member of the cloud strategy team.
    - Refine the list of members to align expected time commitments with expected outcomes.
    - Align on a set of short-term and mid-term metrics to support ongoing progress reports.

* Step 6: Decide whether to proceed or cancel based on the business justification

    Deliverables:
    - Kick off the business justification effort with your supporting teams.
    - Meet with the supporting teams monthly (or as needed) until the strategy team can align on a go/no go decision to proceed with cloud adoption.

* Step 7: Support adoption through a regular cadence

    Deliverables:

    - Prioritization: When the existing digital estate is rationalized, the strategy team helps establish waves of migration or innovation priorities. 
    - Evaluate risks: As cloud adoption grows, new forms of adoption expose new risks. The strategy team is responsible for helping evaluate those new risks. 
    - Review budget and spend: As cloud adoption increases, so will budgets for various workloads in the portfolio. 
    - Business planning: When the adoption teams complete their migration or innovation efforts, additional business planning will be required to maximize return from the new technology solutions. 
    - Executive support: Cloud adoption will result in organizational change. This is most visible within the IT organization.


### cloud adoption team

Cloud adoption teams are the modern-day equivalent of technical implementation teams or project teams. The nature of the cloud might require more fluid team structures.

> Some cloud adoption teams focus exclusively on cloud migration, and others focus on innovations that take advantage of cloud technologies. Some teams include the broad technical expertise that's required to complete large adoption efforts, such as a full datacenter migration, and others have a tighter technical focus. As cloud adoption expands, customers benefit from a team that's dedicated to the cloud platform function.

* Step 1: Determine the type of adoption team you need

    The types of adoption:

    - Migration of existing workloads
    - Modernization of existing workloads and assets
    - Architectural change to existing workloads and assets
    - Development of new workloads

    Deliverables:

    - Determine whether the team aligns better with the Migrate methodology or the Innovate methodology.
    - Each methodology has a four-step onboarding experience to help the team understand the tools and processes required to get really good at that effort. 
    - Align responsibilities across teams by developing a cross-team matrix that identifies responsible, accountable, consulted, and informed (RACI) parties.


* Step 2: Align your team with other supporting teams

    Deliverables:

    - Review design guidance, operational baselines, policies, and processes from the various supporting teams to understand the guardrails that have been established for guiding cloud adoption.
    - Review the guidance with other cloud adoption teams to understand any limitations you might encounter as a result of those guardrails.


* Step 3: Begin your adoption journey

    Deliverables:

    - Become increasingly better at delivering on the methodology associated with your adoption approach.
    - Support other teams in the completion of their accountable steps, even if those steps are blockers to your adoption efforts.

* Step 4: Expand your skills with scenarios and best practices

    Deliverables:
    - Increase skills and experience to address more complex adoption scenarios.


* Step 5: Build a cloud adoption factory

    Deliverables:
    - Improve delivery processes to create a highly optimized adoption factory.



### cloud governance team

A cloud governance team ensures that cloud-adoption risks and risk tolerance are properly evaluated and managed. The team identifies risks that can't be tolerated by the business, and it converts risks into governing corporate policies.


* Step 1: Determine whether a cloud governance team is needed
  
    Deliverables:

    - Determine whether you need a cloud governance team.
    - Align responsibilities across teams by developing a cross-team matrix that identifies responsible, accountable, consulted, and informed (RACI) parties. 

* Step 2: Align with other teams

    Deliverables:

    - Discuss current-state implementation and ongoing adoption plans with each team.

* Step 3: Establish a cadence with other teams

    Deliverables:

    - Establish a cadence with the supporting teams. If possible, align that cadence with release and planning cycles.
    - Establish a separate cadence directly with the cloud strategy team (or various team members) to review risks that are associated with the next wave of adoption and gauge the team's level of tolerance for those risks.

* Step 4: Review the methodology

    Deliverables:

    - Gain an understanding of the methodology, approach, and implementation that supports the Govern methodology.

* Step 5: Complete the governance benchmark

    Deliverables:

    - Complete the governance benchmark assessment, based on conversations with various stakeholders. Or ask other teams to complete the assessment on their own.

* Step 6: Implement the initial governance best practice and configuration

    Deliverables:

    - Deploy the basic governance tools and organization configurations that are required to govern the environment during the next few waves of adoption efforts.

* Step 7: Continuously improve governance maturity

    Deliverables:

    - Implement governance improvements to guard against changing risks and governance needs.

### cloud operations team

An operations team focuses on monitoring, repairing, and remediating issues related to traditional IT operations and assets. In the cloud, many of the capital costs and operations activities are transferred to the cloud provider, giving IT operations the opportunity to improve and provide significant additional value.

* Step 1: Determine whether a cloud operations team is needed

    Deliverables:

    - Determine whether you need a cloud operations team.
    - Align responsibilities across teams by developing a cross-team matrix that identifies responsible, accountable, consulted, and informed (RACI) parties. 

* Step 2: Align with other teams

    Deliverables:

    - Discuss current-state implementation and ongoing adoption plans with each team.

* Step 3: Establish a cadence with other teams

    Deliverables:

    - Establish a cadence with the supporting teams. If possible, align that cadence with release and planning cycles.
    - Establish a separate cadence directly with the cloud strategy team or its various team members to review any operational requirements that are associated with the next wave of adoption.

* Step 4: Review the methodology

    Deliverables:

    - Gain an understanding of the methodology, approach, and implementation that supports the Manage methodology.


* Step 5: Implement the operations baseline

    Deliverables:

    - Deploy the basic Azure server-management configurations that are required for operating the environment during the next few waves of adoption efforts.


* Step 6: Align business commitments

    Deliverables:

    - Document the expectations of business stakeholders.
    - Determine whether advanced operations are required for specific workloads or platforms.

* Step 7: Operations maturity

    Deliverables:

    - Improve operations maturity to support commitments to business stakeholders.

* Step 8: Scale operations consistency through governance

    Deliverables:

    - Help the cloud governance team implement new requirements for resource consistency.

* Step 9: Adoption handoffs

    Deliverables:

    - Regularly review and accept handoffs from cloud adoption teams.


### MVP - team structure

This proven approach is considered a minimum viable product (MVP), because it might not be sustainable. Each team wears many hats, as outlined in the RACI (responsible, accountable, consulted, and informed) charts.

#### Cloud adoption team

This team is accountable for technical solutions, business alignment, project management, and operations for the solutions that are adopted.

#### Cloud governance team
    
To balance the cloud adoption team, a cloud governance team is dedicated to ensuring excellence in the solutions that are adopted. The cloud governance team is accountable for platform maturity, platform operations, governance, and automation.

-               |  Cloud adoption team  | Cloud governance team
--------------------|---------------------|-----------
Solution delivery	| Accountable|   Consulted
Business alignment	| Accountable|   Informed
Change management	| Accountable|   Informed
Solution operations	| Accountable|    Informed
Governance	        |  Consulted |   Accountable
Platform maturity	|  Consulted |   Accountable
Platform operations	|  Consulted |  Accountable
Platform automation  | Informed  | Accountable


- Accountable:	The one person or team to be held accountable for results and outcomes		
- Responsible:	Any people or teams who are responsible for contributing to the results and outcomes		
- Consulted:	People or teams who should be consulted prior to changes being implemented		
- Informed:	People or teams who should be informed about changes		














<file_sep>+++
title = "JS & ES Note - 3"
description="The var , let and const keywords "
+++



### The **var** statement

* var declarations, wherever they occur, are processed before any code is executed. This is called hoisting.

* The scope of a variable declared with var is its current execution context, which is either the enclosing function or, for variables declared outside any function, global. If you re-declare a JavaScript variable, it will not lose its value.

* Assigning a value to an undeclared variable implicitly creates it as a __global__ variable (it becomes a property of the global object) when the assignment is executed. 

> Test yourself with following samples

#### Sample 1

```js
function testVar(){
    console.log(a)
    console.log(b)
    var b = 2
    c=3
}

var a = 1

testVar()

console.log(b)

```
#### Answer of sample 1


```js
function testVar(){
    console.log(a) // 1
    console.log(b) // undefined
    var b = 2
    c = 3
}
var a = 1 
testVar()
console.log(c) // 3
```

#### Sample 2 (Run in browser's console)

```js
d = 13
console.log(this.d)
delete this.d
console.log(this.d)

var e = 31
console.log(this.e)
delete this.e
console.log(this.e)

var i = i + 1
console.log( i )             
console.log( this.i )        
```

#### Answer of sample 2 

```js
var e = 31
f = 13
console.log(this.e, this.f) // 31 13
console.log(e, f)           //  31 13          
delete this.f
delete this.e
console.log(this.e, this.f) // 31 undefinded
console.log(e, f)           //  ReferenceError: d is not defined

var i = i + 1
console.log( i )             // NaN
console.log( this.i )        // NaN
```

* Conclusion: Because of the above unexpected results, it is recommended to always declare variables, regardless of whether they are in a function or global scope. 

> Since the var statement is difficult to harness, people have to come up a solution to address the problem. Then it turns out other new keywords: let & const from ECMAScript 2015 (6th Edition, ECMA-262)

### Let and Const statement

* The let statement declares a block scope local variable, optionally initializing it to a value.

* __let__ allows you to declare variables that are limited to a scope of a block statement, or expression on which it is used, unlike the var keyword, which defines a variable globally, or locally to an entire function regardless of block scope. The other difference between var and let is that the latter is initialized to value only when parser evaluates it (see below).

* Constants are block-scoped, much like variables defined using the let statement. The value of a constant can't be changed through reassignment, and it can't be re-declared.

* The const declaration creates a read-only reference to a value. It does not mean the value it holds is immutable, just that the variable identifier cannot be reassigned. For instance, in the case where the content is an object, this means the object's contents (e.g., its properties) can be altered.

>>> Test yourself with following samples

#### Sample 3

```js
var var1;
let letVar;
const constVar;

function testVar() {
    console.log( var1);
    console.log( constVar);
    console.log( letVar);
}
testVar() 
```

#### Answer of sample 3

```js
var var1;
let letVar;
const constVar; // missing initialization

function testVar() {
    console.log( var1);
    console.log( constVar);
    console.log( letVar);
}
testVar()
```

#### Sample 4

```js

var v1 = "";
var v1 = 123;

let let1 = "";
let let1 = 123;

const c1 = "";
c1 = 123;

```

#### Answer of sample 4



```js
var v1 = "";
var v1 = 123;

let let1 = "";
let let1 = 123; //SyntaxError: Identifier 'let1' has already been declared.

const c1 = "";
c1 = 123; // TypeError: Assignment to constant variable.

```

#### Sample 5 


```js
for ( var i = 0 ; i < 5 ; i++ ){
    var x = 20;
    console.log(i);
}
console.log( i );
console.log( x );

for ( ; i < 10 ; i++ ){
    var i  
    console.log(i);
}
/////////////////////////////////////////


for ( let t = 0 ; t < 5 ; t++ ){
    console.log( t);
    let s = 100
}
console.log(s)  
console.log(t)
```


#### Answer of sample 5



```js
for ( var i = 0 ; i < 5 ; i++ ){
    var x = 20
    console.log(i); // 0 1 2 3 4 
}
console.log( i );   // 5
console.log( x );   // 20

for ( ; i < 10 ; i++ ){
    var i  // re-declare will not reset value
    console.log(i);  // 5 6 7 8 9 
}
/////////////////////////////////////////


for ( let t = 0 ; t < 5 ; t++ ){
    console.log( t); 
    let s = 100
}
console.log(t)  // ReferenceError: t is not defined
console.log(s)  // 

```


<file_sep>+++
title="Pitfalls"
description="Common pitfalls"
weight = 9
+++

### Shadowing

* Hiding (shadowing) a variable by misusing short declaration.

* Such mistakes occur mostly inside the  if-body or for-loop

    ```go
    var remember bool = false
        if something {
            remember := true // Wrong.
        }
    // use remember

    func shadow() (err error) {
        x, err := check1() // x is created; err is assigned to
        if err != nil {
            return // err correctly returned
        }
        
        if y, err := check2(x); err != nil { // y and inner err are created
            return // inner err shadows outer err so nil is wrongly returned!
        } else {
            fmt.Println(y)
        }
        return
    }
    ```


### Misusing strings

* String concatenations of the kind a += b are inefficient, especially when performed inside a loop. 

* Instead one should use a bytes.Buffer to accumulate string content

    ```go
    var b bytes.Buffer
    // ...
    for condition {
        b.WriteString(str) // appends string str to the buffer
    }
    return b.String()
    ```

### Using deffer incorrectly

* Using defer for closing a file in the wrong scope

* It mostly occurs in the for-loop body. Suppose you are processing a range of files in a for-loop, and you want to make sure the files are closed after processing by using defer, 

* BAD defer sample

    ```go
    for _, file := range files {
        if f, err = os.Open(file); err != nil {
        return
        }
                        
        defer f.Close() // This is /wrong/. 
        // The file is not closed when this loop iteration ends.

        // perform operations on f:
        f.Process(data)
    }
    ```

* Defer is only executed at the return of a function, not at the end of a loop or some other limited scope. 

* DO NOT use defer to close the file in the for-loop body

    ```go
    for _, file := range files {
        if f, err = os.Open(file); err != nil {
            return
        }
        // perform operations on f:
        f.Process(data)
        // close f:
        f.Close()
    }
    ```


### Confusing new() and make()

* for slices, maps and channels, use make
* for arrays, structs, and all value types, use new 


### No need for slice

* No need to pass a pointer to a slice to a function

* A slice is a pointer to an underlying array. Passing a slice as a parameter to a function is probably what you always want: namely passing a pointer to a variable to be able to change it, and not passing a copy of the data.

* Do not dereference a slice when used as a parameter!

    ```go
    // correct way
    func findBiggest( listOfNumbers []int ) int {}
    // wrong way 
    func findBiggest( listOfNumbers *[]int ) int {}
    ```


###  Using pointers to interface

* Look at the following program: nexter is an interface with a method next() meaning read the next byte. nextFew1 has this interface type as parameter and reads the next num bytes, returning them as a slice: this is ok. 

    ```go
    package main
    import (
       "fmt"
    )
    type nexter interface {
      next() byte
    }
    func nextFew1(n nexter, num int) []byte {
        var b []byte
        for i:=0; i < num; i++ {
            b[i] = n.next()
        }
       return b
    }
    func nextFew2(n *nexter, num int) []byte {
        var b []byte
        for i:=0; i < num; i++ {
            b[i] = n.next() // compile error:
            // n.next undefined (type *nexter has no field or method next)
        }
        return b
    }
    func main() {
      fmt.Println("Hello World!")
    }
    ```

### Misusing pointers

* Passing a value as a parameter in a function or as receiver to a method may seem a misuse of memory, because a value is always copied. But on the other hand values are allocated on the stack, which is quick and relatively cheap. 

* If you would pass a pointer to the value instead the Go compiler in most cases will see this as the making of an object, and will move this object to the heap, so also causing an additional memory allocation: therefore nothing was gained in using a pointer instead of the value!


### Misusing goroutines and channel

* In practice often you don’t need the concurrency, or you don’t need the overhead of the goroutines with channels, passing parameters using the stack is in many cases far more efficient.

* Moreover it is likely to leak memory if you break or return or panic your way out of the loop, because the goroutine then blocks in the middle of doing something. In real code, it is often better to just write a simple procedural loop. Use goroutines and channels only where concurrency is important!


    ```go
    var values = [5]int{10, 11, 12, 13, 14}

    func main() {
        // version A:
        fmt.Println("\nVersion A:")
        for ix := range values { // ix is the index
            func() {
                fmt.Print(ix, " ")
            }() // call closure, prints each index
        }
        fmt.Println()
        // version B: same as A, but call closure as a goroutine
        fmt.Println("\nVersion B:")
        for ix := range values {
            go func() {
                fmt.Print(ix, " ")
            }()
        }
        fmt.Println()
        time.Sleep(5e9)
        // version C: the right way
        fmt.Println("\n\nVersion C:")
        for ix := range values {
            go func(ix interface{}) {
                fmt.Print(ix, " ")
            }(ix)
        }
        fmt.Println()
        time.Sleep(5e9)
        // version D: print out the values:
        fmt.Println("\n\nVersion D:")
        for ix := range values {

            val := values[ix]
            go func() {
                fmt.Print(val, " ")
            }()
        }
        time.Sleep(1e9)
    }

    //----- output -------------
    // Version A:
    // 0 1 2 3 4 

    // Version B:
    // 4 4 4 4 4 

    // Version C:
    // 1 3 4 0 2 

    // Version D:
    // 14 10 13 12 11
    ```


### Bad error handling

#### Don’t use booleans:

* Making a boolean variable whose value is a test on the error-condition like in the following is superfluous

```go
var good bool
// test for an error, good becomes true or false
if !good {
    return errors.New(“things aren’t good”)
}
////--------------
//... 
err1 := api.Func1()
if err1 != nil { … }
```


#### Don’t clutter your code with error-checking

* BAD sample. 

    ```go
    // ... 
    err1 := api.Func1()
    if err1 != nil {
        fmt.Println(“err: “ + err.Error())
        return
    }
    err2 := api.Func2()
    if err2 != nil {
        //...
        return
    }
    ```

* With the above pattern, it is hard to tell what is normal program logic and what is error checking/reporting. Also notice that most of the code is dedicated to error conditions at any point in the code. A good solution is to wrap your error conditions in a closure wherever possible

    ```go
    err := func () error {
        if req.Method != “GET” {
        return errors.New(“expected GET”)
        }
        if input := parseInput(req); input != “command” {
        return errors.New(“malformed command”)
        }
        // other error conditions can be tested here
    } ()
    if err != nil {
        w.WriteHeader(400)
        io.WriteString(w, err)
        return
    }
    ```



























<file_sep>+++

title = "Java Note - 1: Enum"
description="Replace constant property of the interface or abstract class with Enum"
+++


## Prerequisites

>*Java 1.5+*

## New type: Enum

Enum was a great improvement in Java 1.5. From that more and more developer abandom the interface or abstract class as constant variable container. 

### Before Java 5

Before Java 1.5 you will following coding in many Java program. 

```java
    // Use interface or abstract class as constant variable container
    public interface Country {
         public static final String AU = "Australian";
         public static final String UK = "United Kingdom"; 
         public static final String US = "United State"; 
    }
    
    public class Util {
         
        public static String getLanguageCode(String country) {
    
            String languageCode = "en";
            switch (country) {
                case Country.AU:
                    languageCode = "en-au";
                    break;
                case Country.UK:
                    languageCode = "en-uk";
                    break;
                case Country.US:
                    languageCode = "en-us";
                    break;
            }
            return languageCode;
        }
    }
```

Above program looks very good. Please take a close look and check it carefully. You will find the program will never return __*en-au*__, since there is a typo in the constant AU. It should be __*Australia*__ instead of __*Australian*__. I believe many developers have short sight problem like me, and it happened again and again. Using string as constant flag is not a good option, but there is no other better solution before Java 1.5.

### After Java 5

After Java 1.5, you will see the change below. Enum is the best container for constants. It can help you check the program time. Meanwhile, it can simplfy your coding. 



```java
// Use Enum as constant variable container
public enum  Country {
    Australia, UnitedKingdom, UnitedState
}

public class Util {
        
    public static String getLanguageCode(Country country) {

        String languageCode = "en";
        switch (country) {
            case Australia:
                languageCode = "en-au";
                break;
            case UnitedKingdom:
                languageCode = "en-uk";
                break;
            case UnitedState:
                languageCode = "en-us";
                break;
        }
        return languageCode;
    }
}
```

Now you program will not be by any unintentional typo, since it will throw you compile error before you run the application. If you haven't refactor your static constants container, it is time to improve it now. 

After Java Java 1.7, there are some new features. One of these new features is Switch statement. Now it supports String. It is a great for Java developer.With this new feature, the old Enum can be enhanced and the Util class can provide more handy methods (Overload method getLanguageCode) for development. 

New Enum class can support flexible requirement. In the early version of Enum, the toString method only will return exactly the specified constanct name. Now it can be overrided with toString to return different constant name. 

```java
// It can return customized name and simplify coding 
public enum Country {
    AU("Australia", "au","en-au"), 
    UK("United Kingdom", "en-uk"),
    US("United State", "us","en-us");

    String countryName;
    String countryCode;
    String languageCode;
    
    private Country(String name, String code) {
        countryName = name;
        countryCode = code;
    }

    public String getCode() {
        return countryCode;
    }
    
    public String getLanguageCode() {
        return languageCode;
    }
    
    @Override
    public String toString() {
        return countryName;
    }
}
```

The Uitl class can convert any country name or country code to Enum Country, vice versa. Now developer can seamless convert the String from UI to the Enum, since on the UI, usually the country name will Australia or United Kingdom instead of just AU or UK. For coding, use AU or UK can simplify coding and is friendly to developer. 

```java
public class Util {
        
    public static Country convertCountryNameOrCode(String nameOrCode ) {
        
        Country country = null;
        switch (nameOrCode) {
            case "au":
            case "AU":
            case "Australia":
                country = Country.AU;
                break;
            case "uk":
            case "UK":    
            case "United Kingdom":
                country = Country.UK;
                break;
            case "us":
            case "US":    
            case "United State":
                country = Country.UK;
                break;
        }
        return country;
    }
    
    public static String getCountryName(  Country country ){
        String countryName = null;
        switch (country){
            case AU:
                countryName = Country.AU.toString();
                break;
            case US:
                countryName = Country.UK.toString();
                break;
            case UK:
                countryName = Country.US.toString();
                break;
            default:
                System.err.println("Unknow Country");
                assert false;
                break;
        }
        return countryName;
    }

    
    public static String getCountryCode(  Country country ){
        String countryCode = null;
        switch (country){
            case AU:
                countryCode = Country.AU.getCode();
                break;
            case US:
                countryCode = Country.UK.getCode();
                break;
            case UK:
                countryCode = Country.US.getCode();
                break;
            default:
                System.err.println("Unknow Country");
                assert false;
                break;
        }
        return countryCode;
    }
        
    public static String getLanguageCode(Country country) {
    
            String languageCode = "en";
            switch (country) {
                case AU:
                    languageCode = Country.AU.getLanguageCode() ;
                    break;
                case UK:
                    languageCode =  Country.UK.getLanguageCode();
                    break;
                case US:
                    languageCode =  Country.US.getLanguageCode();
                    break;
            }
            return languageCode;
    }
}
```
<file_sep>+++
title = "TS: Basic Types"
description="Basic Types"
+++

## Basic Types

In TypeScript, the same types as you would expect in JavaScript are supported, with a convenient enumeration type thrown in to help things along.

### Types in JavaScript

* Boolean - The most basic datatype is the simple true/false value, aka boolean value.

        let isDone: boolean = false;

* Number -  All numbers in TypeScript are floating point values. 

        let decimal: number = 6;
        let hex: number = 0xf00d;
        let binary: number = 0b1010;
        let octal: number = 0o744;

* String - Another fundamental part of creating programs in JavaScript for webpages and servers alike is working with textual data. TypeScript also uses double quotes (") or single quotes (') to surround string data.

        let color: string = "blue";
        color = 'red';

* Template strings - It  can span multiple lines and have embedded expressions. These strings are surrounded by the backtick/backquote (`) character, and embedded expressions are of the form ${ expr }.

    ```ts
    let sentence: string = `Hello, my name is ${ fullName }.

    I'll be ${ age + 1 } years old next month.`;
    ```

* Array - Two ways, the elements followed by [] to denote an array of that element type; The second way uses a generic array type, Array<elemType>. 

        let list: number[] = [1, 2, 3];
        let list: Array<number> = [1, 2, 3];

* Object - object is a type that represents the non-primitive type, i.e. anything that is not number, string, boolean, bigint, symbol, null, or undefined.

        declare function create(o: object | null): void;

        create({ prop: 0 }); // OK
        create(null); // OK

        create(42); // Error
        create("string"); // Error
        create(false); // Error
        create(undefined); // Error



### Types in Typescdript 

* Tuple - Tuple types allow you to express an array with a fixed number of elements whose types are known, but need not be the same. 

        let x: [string, number, boolean];
        x=['text', 1, true]; // OK
        x=['text', , true]; // Error, Type 'undefined' is not assignable to type 'number'.
        x=['text', 0, 0]; // Error, Type 'number' is not assignable to type 'boolean'.
        console.log(x[0]); // OK 
        console.log(x[3]); // Error, Tuple type '[string, number, boolean]' of length '3' has no element at index '3'.

* Enum - A helpful addition to the standard set of datatypes from JavaScript is the enum. By default, enums begin numbering their members starting at 0. You can change this by manually setting the value of one of its members. 


        enum Color {Red, Green, Blue}
        let c: Color = Color.Green;
        console.log(c);  // output: 0


        enum Color2 { Red = 1, Green = 2, Blue = 4 }
        let d: Color2 = Color2.Green;
        console.log(d);   // output: 2


        enum Color3 { Red = 10, Green , Blue }
        let e: Color3 = Color3.Blue;
        console.log(e);  // output: 12

* Any - The any type is a powerful way to work with existing JavaScript, allowing you to gradually opt-in and opt-out of type checking during compilation. You might expect Object to play a similar role, as it does in other languages. However, variables of type Object only allow you to assign any value to them. You can’t call arbitrary methods on them, even ones that actually exist.


        let a: any = 4.001;
        console.log(a.toFixed()) // output: 4

        let o: Object = 4.001;
        console.log(o.valueOf()) // output: 4.001
        console.log(o.toFixed()) // Error - Property 'toFixed' does not exist on type    'Object'.


* Void - void is a little like the opposite of any: the absence of having any type at all. Declaring variables of type void is not useful because you can only assign null (only if --strictNullChecks is not specified, see next section) or undefined to them.

        function warnUser(): void {
            console.log("This is my warning message");
        }

        let unusable: void = undefined;
        unusable = null; // OK if `--strictNullChecks` is not given



* Null and Undefined - both undefined and null actually have their own types named undefined and null respectively.When using the --strictNullChecks flag, null and undefined are only assignable to any and their respective types (the one exception being that undefined is also assignable to void). This helps avoid many common errors. In cases where you want to pass in either a string or null or undefined, you can use the union type string | null | undefined.

        let u: undefined = undefined;
        let n: null = null;


* Never - The never type represents the type of values that never occur. For instance, never is the return type for a function expression or an arrow function expression that always throws an exception or one that never returns; Variables also acquire the type never when narrowed by any type guards that can never be true. The never type is a subtype of, and assignable to, every type; however, no type is a subtype of, or assignable to, never (except never itself). Even any isn’t assignable to never.

        function error(message: string): never {
            throw new Error(message);
        }

        // Function returning never must have unreachable end point
        function infiniteLoop(): never {
            while (true) {
            }
        }







<file_sep>+++
title = "Azure: RBAC - 3"
weight = 1
description="Best practices & Azure AD Privileged Identity Management"

+++

## Best practices 

### Only grant the access users need

Scope | Roles   | | | | |
------|--------|-------------------|--------|-------------|----|
-     | Reader | Resource-specific | Custom | Contributor | Owner|
Management Group | Observers | Users managing resources| Users managing resources| Users managing resources | Admins
Subscriptions | Observers | Users managing resources |  Users managing resources | Users managing resources| Admins
Resource Group | Observers | Users managing resources |  Users managing resources | Users managing resources |  Admins
Resources | Automated processes | Automated processes | Automated processes | Automated processes | Automated processes | 


### Azure AD Privileged Identity Management

Privileged Identity Management (PIM) is a service in Azure Active Directory (Azure AD) that enables you to manage, control, and monitor access to important resources in your organization. These resources include resources in Azure AD, Azure, and other Microsoft Online Services such as Microsoft 365 or Microsoft Intune.


#### What does it do

Privileged Identity Management provides time-based and approval-based role activation to mitigate the risks of excessive, unnecessary, or misused access permissions on resources that you care about.

- Provide just-in-time privileged access to Azure AD and Azure resources
- Assign time-bound access to resources using start and end dates
- Require approval to activate privileged roles
- Enforce multi-factor authentication to activate any role
- Use justification to understand why users activate
- Get notifications when privileged roles are activated
- Conduct access reviews to ensure users still need roles
- Download audit history for internal or external audit

<file_sep>+++
title = "PostgresQL Note - 1"
description="Introduction of SQL for PostgresQL"
+++


> PostgreSQL is a powerful, open source object-relational database system with over 30 years of active development that has earned it a strong reputation for reliability, feature robustness, and performance. 

### Getting Started

* Switch to user postges

```bash
sudo su - postgres

psql
```

* Create new login id as super admin

It is frequently convenient to group users together to ease management of privileges: that way, privileges can be granted to, or revoked from, a group as a whole. In PostgreSQL this is done by creating a role that represents the group, and then granting membership in the group role to individual user roles.

> In the SQL standard, there is a clear distinction between users and roles, and users do not automatically inherit privileges while roles do. This behavior can be obtained in PostgreSQL by giving roles being used as SQL roles the INHERIT attribute, while giving roles being used as SQL users the NOINHERIT attribute. However, PostgreSQL defaults to giving all roles the INHERIT attribute, for backward compatibility with pre-8.1 releases in which users always had use of permissions granted to groups they were members of.

> The role attributes LOGIN, SUPERUSER, CREATEDB, and CREATEROLE can be thought of as special privileges, but they are never inherited as ordinary privileges on database objects are. You must actually SET ROLE to a specific role having one of these attributes in order to make use of the attribute. 

```sql
CREATE ROLE user_id WITH
LOGIN
  SUPERUSER
  CREATEDB
  CREATEROLE
  INHERIT
  REPLICATION
  CONNECTION LIMIT -1
  PASSWORD '<PASSWORD>';
  
GRANT postgres, pg_monitor, pg_read_all_settings, 
pg_read_all_stats, pg_signal_backend, pg_stat_scan_tables 4
TO hho WITH ADMIN OPTION;

```


* Change password

```sql
ALTER USER user_id WITH PASSWORD '<PASSWORD>';
```

* Once the group role exists, you can add and remove members using the GRANT and REVOKE commands

```sql
GRANT postgres TO new_role;
REVOKE postgres FROM new_role;
```

* Create new database with new login id and load sql script to initialize the database

```sql
sudo su - user_id psql -c 'createdb db_01;'

sudo su - user_id psql  -d postgres -f init_db.sql
```

* Grant the privilege of database __db_01__ to other user

```sql
 GRANT ALL PRIVILEGES ON DATABASE db_01 to another_user;
```

<file_sep>+++
title = "F#"
description = "F# Tutorials"
+++

{{%children style="card" description="true" sort="Weight" %}}<file_sep>+++
title = "Docker Practices"
description="Useful & practical docker practices"
+++

> Docker is an open platform for developing, shipping, and running applications. Docker provides the ability to package and run an application in a loosely isolated environment called a container. The isolation and security allow you to run many containers simultaneously on a given host. Containers are lightweight because they don’t need the extra load of a hypervisor, but run directly within the host machine’s kernel. This means you can run more containers on a given hardware combination than if you were using virtual machines. You can even run Docker containers within host machines that are actually virtual machines!

> Docker enables you to separate your applications from your infrastructure so you can deliver software quickly. With Docker, you can manage your infrastructure in the same ways you manage your applications. By taking advantage of Docker’s methodologies for shipping, testing, and deploying code quickly, you can significantly reduce the delay between writing code and running it in production.


## Common Use Cases

### Test with busybox image

```
docker run -it --rm busybox echo Hello World
## output 
Hello World
```

### Test with nginx image

Create a test web page with content below

```html
<!DOCTYPE html>
<html >
<head>
   <title>Docker Nginx</title>
</head>
<body>
    <h2>Hello from docker</h2>
</body>
</html>
```

Test the web page with ngnix image

```
docker run -it --rm -d -p 8080:80 --name web -v ~/app:/usr/share/nginx nginx
```

### build 

Build an image from a Dockerfile. The presence of Dockerfile is mandatory. The file name convention is __Dockerfile__ or **your_customized_filename.Dockerfile**

    $ docker build <path_of_workspace>

Build an image with tag

    $ docker build <path_of_workspace> -t <image_tag>

Build an image with specific dockerfile

    $ docker build <path_of_workspace> -f  <path_of_dockerfile>

### tag or untag

    $ docker image tag SOURCE_IMAGE[:TAG] TARGET_IMAGE[:TAG]
    
    $ docker rmi [unwanted_tag]
### push 

    $ docker push <docker_repo>:<image_tag>

### scripting

The build & push can be simplified with  some scripting. Here I recap a script from my other docker repository below.

```bash
DOCKER_REPO=$1

if [ -z "$1" ]; then 
    DOCKER_REPO='harryh00/docker-kits'
fi

echo docker repo: $DOCKER_REPO

############################################################################
## --------------------- Build images  ---------------------------
############################################################################

build_image() {
    FOLDER=$1
    echo "----- Build ${FOLDER} based image -----"
    for fname in $(ls ${FOLDER}); do
        PREFIX=${fname/".Dockerfile"/""}
        echo ":::: Build ${DOCKER_REPO}:${FOLDER}-${PREFIX}"
        docker build ${FOLDER} -f "${FOLDER}/${PREFIX}.Dockerfile" \
            -t "${DOCKER_REPO}:${FOLDER}-${PREFIX}"
    done
}

##############################################################################
## --------------------- Push images to docker hub ---------------------------
##############################################################################

## alpine basealpined image
push_image() {
    FOLDER=$1
    echo "----- Push ${FOLDER} based image -----"
    for fname in $(ls ${FOLDER}); do
        PREFIX=${fname/".Dockerfile"/""}
        echo ":::: Push ${DOCKER_REPO}:${FOLDER}-${PREFIX}"
        docker push ${DOCKER_REPO}:${FOLDER}-${PREFIX}
    done
}

# Main
main() {
    docker login
    docker info
    FOLDERS=(
        alpine
        ubuntu
        centos
    )
    for FOLDER in ${FOLDERS[@]}; do
        build_image ${FOLDER}
        push_image ${FOLDER}
    done
}

main "$@"
```

### 2-step build

For production deployment, usually we just deploy the delivery instead of the full copy of source code. To achieve that, we can the build into 2 steps. The first step is to build the source code. and the second one is to build the deliverable image. 

I recap one docker file from the my repository vue-crm here


```dockerfile
###### Build the App #####
FROM node:10.19 AS node
LABEL author="<NAME>"
WORKDIR /
COPY . .
RUN npm install
RUN npm run build -- --prod


###### Build the Delivery #####
FROM nginx:alpine
LABEL author="Harry Ho"
WORKDIR /var/cache/nginx
COPY --from=node /dist /usr/share/nginx/html
COPY ./config/nginx.conf /etc/nginx/conf.d/default.conf

```


<file_sep>FROM nginx:alpine
LABEL author="<NAME>"
COPY   ./public /usr/share/nginx/html




<file_sep>+++
date = "2021-12-20T14:59:31+11:00"
title = "Ubuntu Desktop 20 LTS note"
description = "Post-installation for Ubuntu 20 desktop"
+++

## Ubuntu 20.04.3 LTS (Focal Fossa)

Ubuntu is the world’s most popular open-source desktop operating system. Ubuntu 20.04 LTS is an enterprise-grade, secure, cost-effective operating system for organisations and home users. 



### Zsh Prezto

#### Install Prezto


```sh
clear
sudo apt-get install -y git
sudo apt-get update && sudo apt-get install -y zsh
# Get prezto
git clone --recursive https://github.com/sorin-ionescu/prezto.git ~/.zprezto

# Backup zsh config if it exists
if [ -f ~/.zshrc ];
    then
        mv ~/.zshrc ~/.zshrc.backup
fi

# Create links to zsh config files
ln -s ~/.zprezto/runcoms/zlogin ~/.zlogin
ln -s ~/.zprezto/runcoms/zlogout ~/.zlogout
ln -s ~/.zprezto/runcoms/zpreztorc ~/.zpreztorc
ln -s ~/.zprezto/runcoms/zprofile ~/.zprofile
ln -s ~/.zprezto/runcoms/zshenv ~/.zshenv
ln -s ~/.zprezto/runcoms/zshrc ~/.zshrc

```

#### Change theme & module

* Update the theme 'sorin' to 'steeef' in .zpreztorc
* Add following plugins

```
zstyle ':prezto:load' pmodule \
  'environment' \
  'terminal' \
  'editor' \
  'history' \
  'directory' \
  'spectrum' \
  'utility' \
  'completion' \
  'git' \
  'syntax-highlighting' \
  'history-substring-search' \
  'prompt'

```

#### Change shell to Zsh

```sh
chsh -s $(Which zsh)
source ~/.zshrc
```

### Install docker


#### Set up the repository


```sh
 sudo apt-get update

 sudo apt-get install \
    ca-certificates \
    curl \
    gnupg \
    lsb-release
```

#### Add Docker’s official GPG key:

```sh
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor \
    -o /usr/share/keyrings/docker-archive-keyring.gpg

# Use the following command to set up the stable repository. 
# To add the nightly or test repository, add the word nightly or test (or both) 
# after the word stable in the commands below. Learn about nightly and test channels.

echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
```



#### Install Docker Engine

```sh
# Update the apt package index, and install the latest version of 
# Docker Engine and containerd, or go to the next step to install 
# a specific version:

 sudo apt-get update

 sudo apt-get install docker-ce docker-ce-cli containerd.io
```


#### Post installation of Docker

```sh

# Create the docker group.
sudo groupadd docker

# Add your user to the docker group.
sudo usermod -aG docker $USER

# Log out and log back in so that your group membership is re-evaluated.

# On Linux, you can also run the following command to activate the changes to groups:

newgrp docker

# Test the docker
docker ps
```

#### Add Docker completion to Zsh

* Add the completion to prezto

```sh
curl -fLo ~/.zprezto/modules/completion/external/src/_docker \
https://raw.githubusercontent.com/docker/cli/master/contrib/completion/zsh/_docker

```

* Add following line to zshrc

```
autoload -Uz compinit; compinit
```


### Git

#### Set git credential store


```sh
git config credential.helper store	
# OR 
git config --global credential.helper store	
```


### Dotnet Core 6 SDK

*  Add the Microsoft package signing key to your list of trusted keys and add the package repository.

```sh

wget https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
rm packages-microsoft-prod.deb

```

* Install the SDK

```sh
sudo apt-get update; \
  sudo apt-get install -y apt-transport-https && \
  sudo apt-get update && \
  sudo apt-get install -y dotnet-sdk-6.0
```


### Golang 


* Download & install the golang tal ball

```sh
# Download the linux tar ball from golang site
sudo rm -rf /usr/local/go && sudo tar -C /usr/local -xzf go1.17.5.linux-amd64.tar.gz
```


* Set the Go to PATH on file `.zshrc`

```
export PATH=$PATH:/usr/local/go/bin
```

* Verify the version 

```
go version
```



### Rustlang 

* Use universal script

```
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```


### Java: OpenJDK

* Install LTS JDK

```sh
 sudo apt install -y openjdk-8-jdk \
    openjdk-11-jdk \
    openjdk-17-jdk

```

* Setup default JDK

```sh
sudo update-alternatives --config java
```

<file_sep>+++
title = "Qemu & Virtual Machine"
description="Use Qemu to create virtual machines"
+++

___What is Qemu ?___

[QEMU](https://wiki.qemu.org/Main_Page)  is free and open source. And is licensed under GPL 2. it has the ability to run under both KVM and XEN models (if you enabled virtualization technology from your BIOS first) and offers a lot of options and virtualization options. In this article, we’ll explain how to use QEMU and install it.

QEMU is a virtualization technology emulator that allows you to run operating systems and Linux distributions easily on your current system without the need to install them or burn their ISO files. It is like VMware or VirtualBox. You can use it at anytime to emulate running any operating system you want on a lot of devices and architecture.


## Prerequisites

* Install [Qemu](https://www.qemu.org/download/#linux) on Linux (Ubuntu / Cent OS) or MacBook 
* Create a folder to store the images files. e.g. ~/ws/vms

## Assumptions
* Assume the new iso files are stored in folder "Download"
* Architect of your machine is x86_64
* Install Ubuntu as VM on Qemu

## Create new image


### Create a new image 

Use `qemu-img` to create an image file with a maximum size of 10GB

    qemu-img create -f qcow2 ~/ws/vms/ubuntu16-vm.img 10G


### Resize the maximum

Resize the maximum of the existing image by adding 30GB

    qemu-img resize ~/ws/vms/ubuntu16-vm.img +30G


### Create new vm 

Create a new VM (virtual machine) on the image file

    qemu-system-x86_64 -m 2048 -boot d -enable-kvm \
    -smp 2 -net nic -net user \
    -hda ~/ws/vms/ubuntu16-vm.img \
    -cdrom ~/Downloads/ubuntu-16.04.iso


`-m 2048`: Here we chose the RAM amount that we want to provide for QEMU when running the ISO file. We chose 2048MB here. You can change it if you like according to your needs.

`-boot -d`: The boot option allows us to specify the boot order, which device should be booted first? `-d` means that the CD-ROM will be the first, then QEMU will boot normally to the hard drive image. We have used the `-cdrom`: option as you can see at the end of the command. You can use `-c` if you want to boot the hard drive image first.

`-enable-kvm`: This is a very important option. It allows us to use the KVM technology to emulate the architecture we want. Without it, QEMU will use software rendering which is very slow. That’s why we must use this option, just make sure that the virtualization options are enabled from your computer BIOS.

`-smp 2`: If we want to use more than 1 core for the emulated operating system, we can use this option. We chose to use 2 cores to run the virtual image which will make it faster. You should change this number according to your computer’s CPU.

`-net nic -net user`: By using these options, we will enable an Ethernet Internet connection to be available in the running virtual machine by default.

`-hda testing-image.img`: Here we specified the path for the hard drive which will be used. In our case, it was the testing-image.img file which we created before.

`-cdrom ubuntu-16.04.iso`: Finally we told QEMU that we want to boot our ISO file “ubuntu-16.04.iso”.



### Start the new VM

      qemu-system-x86_64 -m 2048 -boot d -enable-kvm \
        -smp 2 -net nic -net user \
        -hda ~/ws/vms/ubuntu16-vm.img 

### Post-Create VM

* Install `spice-vdagent` to Copy content from Host to Guest
* Install `tmux` 


<file_sep>+++
date = "2016-12-07T16:56:21+11:00"
title = "Tensorflow Note - 1"
description = "Tensorflow Note - 1"

+++

> TensorFlow is an end-to-end open source platform for machine learning. It has a comprehensive, flexible ecosystem of tools, libraries and community resources that lets researchers push the state-of-the-art in ML and developers easily build and deploy ML powered applications.

### Prerequisites

* You are using Ubuntu 16
* Your machine has Nvidia GPU card installed
* If you are using Ubuntu 14, the installation of CUDA and cuDNN will be a bit different. Please check Google's instructions. 


### Install python3 and pip3

* [Please find instructions here](https://harryho.github.io/os/ubuntu-server-14)

### Install virtualenv via pip3 
    
    pip3 install virtualenv

### Create two tensorflow virtualenvs.

    mkdir ~/.envs
    virtualenv --system-site-packages ~/.envs/tf  ### CPU only
    virtualenv --system-site-packages ~/.envs/tfgpu   ### GPU enabled

### Install tensorflow for different virtualenvs

    source ~/.envs/tf/bin/activate
    source ~/.envs/tfgpu/bin/activate
    
    pip3 install tensorflow ### CPU only
    pip3 install tensorflow-gpu ### GPU enabled

### Install CUDA and cuDNN for tensorflow-gpu


* Use following command to check you GPU information

    ```bash
    lspic -nn | grep '\[03' ]
    lshw -numeric -C display

    ### GPU info sample
    ### NVIDIA Corporation GM107M [GeForce GTX 850M]
    ```

* Download and install Nvidia driver based on above [GPU info](http://www.geforce.com/drivers)

    ```bash
    chmod +x NVIDIA-Linux-xxxx.run
    sudo ./NVIDIA-Linux-xxxx.run
    ``` 

* Download and install [CUDA from NVIDIA](https://developer.nvidia.com/cuda-downloads)

    ```bash
    sudo dpkg -i cuda-repo-xxxxx.deb
    sudo apt-get udpate
    sudo apt-get install cuda
    ```

* Setup CUDA_HOME 

    ```bash
    ### CUDA
    export CUDA_HOME=/usr/local/cuda-8.0 
    export LD_LIBRARY_PATH=${CUDA_HOME}/lib64
    ``` 


* Download and install [cuDNN for CUDA](https://developer.nvidia.com/cudnn)

    ```bash
    ### extra the cuDNN tar ball
    tar -xvf cudnn-8.0
    cd cuda 
    sudo cp lib64/* /usr/local/cuda-8.0/lib64
    sudo cp include/* /usr/local/cuda-8.0/include
    ```

### Use sample code to test Tensorflow

* Save code below to file test.py 

    ```python
    import numpy as np
    import tensorflow as tf

    ### Model parameters
    W = tf.Variable([.3], tf.float32)
    b = tf.Variable([-.3], tf.float32)
    ### Model input and output
    x = tf.placeholder(tf.float32)
    linear_model = W * x + b
    y = tf.placeholder(tf.float32)
    ### loss
    loss = tf.reduce_sum(tf.square(linear_model - y)) ### sum of the squares
    ### optimizer
    optimizer = tf.train.GradientDescentOptimizer(0.01)
    train = optimizer.minimize(loss)
    ### training data
    x_train = [1,2,3,4]
    y_train = [0,-1,-2,-3]
    ### training loop
    init = tf.global_variables_initializer()
    sess = tf.Session()
    sess.run(init) ### reset values to wrong
    for i in range(1000):
    sess.run(train, {x:x_train, y:y_train})

    ### evaluate training accuracy
    curr_W, curr_b, curr_loss  = sess.run([W, b, loss], {x:x_train, y:y_train})
    print("W: %s b: %s loss: %s"%(curr_W, curr_b, curr_loss))
    ```

* Test with tensorflow-gpu (GPU enabled)

    ```bash
    source ~/.envs/tfgpu/bin/activate
    python3 test.py

    ## You will probably see the result as follow 
    ## ....
    ## name: GeForce GTX 850M
    ## major: 5 minor: 0 memoryClockRate (GHz) 0.9015
    ## pciBusID 0000:0a:00.0
    ## Total memory: 3.95GiB
    ## Free memory: 3.58GiB
    ## 2017-04-25 10:25:59.640621: I tensorflow/core/common_runtime/gpu/gpu_device.cc:908] DMA: 0 
    ## 2017-04-25 10:25:59.640626: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 0:   Y 
    ## 2017-04-25 10:25:59.640640: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] 
    ### Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 850M, pci ## bus id: 0000:0a:00.0)
    ## W: [-0.9999969] b: [ 0.99999082] loss: 5.69997e-11
    ```









<file_sep>+++

title = "Python Web Framework"
description = "Introduction of Django: Most popular python web framework"
+++

> Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design. Built by experienced developers, it takes care of much of the hassle of Web development, so you can focus on writing your app without needing to reinvent the wheel. It’s free and open source. 

## Install Python, pip and virtualenv

* Windows: Please find in from Home Page

* Ubuntu: Please find it from [Ubuntu setup](/os/ubuntu-server-14)

* Python 3.4 (released March 2014) and Python 2.7.9 (released December 2014) ship with Pip.

* You can simply use pip or pip3 install any package you need.




## **Windows**

### Install django

* create a folder virtualenvs within the location of python 3

* create a new virtualenv named django18

* Activate the new virtual env

* Install Django 1.x.x (LTS version) 

```bash
cd  /path/to/python3
cd  virtualenvs

virtualenv django18

cd django18

Scripts\activate

which python

pip install django==1.x.x
```

### Create django project 

* SET PATH in current command promp

* Navigate to workspace folder

* Create new django project

* Start the app

    ```bat
    SET PATH=c:\apps\python3\virtualenvs\django18\Script;%PATH%

    which django-admin

    cd c:\ws\python\django\

    django-admin startproject demo

    cd demo

    python manage.py runserver
    ```

### Create a new app module

```bash
python manage.py startapp main
```

### Create a new db 

__following commands are just tested in django 1.8__

```python
python manage.py syncdb 
python manage.py makemigrations new_app 
## migrate 
pyrhon manage.py sqlmigrate new_app 0001 
## migrate 
python manage.py migrate new_app 0001
```

### use python shell Model API

```python
python manage.py shell
>>> from XXX.models import ModelClass
>>> ModelClass.objects.all()
>>> ModelClass.objects.get(pk =1 )
>>> ModelClass.objects.filter( fieldName1="abc")
>>> mc = new ModelClass.( fieldName1 = "abc", fieldName2="def", fieldName3 = 3 )
>>> mc.save()
>>> mc = ModelClass.objects.get(pd=1)
>>> mc.delete()    
```

## **Linux**

```bash
pip install virtualenv

pip3 install virtualenv

cd ~

mkdir .envs

## create python2 env
virtualenv -p /usr/bin/python2.7 py2env

virtualenv -p /usr/bin/python3.4 py3env

cd py2env 
source bin/activate

## Check python path 
which python

## Exit
deactivate

```

**Activate virtual environment need to use source instead of executing sh file**

<file_sep>+++


title = "Kubernetes Cluster in 5min"
description="5 Mins to create a kubernetes cluster on Ubuntu or Centos Linux machine"

+++

## Kubernetes

> Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications.

> It groups containers that make up an application into logical units for easy management and discovery. Kubernetes builds upon 15 years of experience of running production workloads at Google, combined with best-of-breed ideas and practices from the community.


## Purpose

This project focuses on the training and demonstration. Please DO NOT use it in production environment.

## Screenshot of kubernetes dashboard

* Below is snapshot of What you'll get after completing all steps

> ![kube cluster dashboard](/img/kube_pods_metrics.jpg)



## Prerequisites


* You have a Linux machine in place. Physical or virtual machine doesn't mather.

* The Linux OS is supposed to be one of following distroes: the Ubuntu 16+, 18+ or CentOS 7.

* Internet is available on your machine

 
## Caveat

* Use Virtual Machine to test it before running it on physical machine

* VirtualBox or VMWare is a good option.

* Kubernetes is supposed to run on Linux server, but it should be able to run on desktop version as well. 


## Steps to use the script 

* Setup SSH Server if you need

* Get the script from my github repo

* Switch to root user and run the script

    ```bash
    # Ubuntu
    wget https://github.com/harryho/kube-cluster-in-5mins/blob/master/ubuntu/kube-cluster.sh

    # CentOS
    wget https://github.com/harryho/kube-cluster-in-5mins/blob/master/centos/kube-cluster.sh

    # Switch user
    sudo su
    ./kube-cluster.sh -h

    ```


## Trouble shooting

* Hardware ( Enable VT-x,  2+ Core , 2G RAM )
* The scripts only support ubuntu 16+ or CentOS 7+
* Swap must be off

    * disable swap immediately

    ```bash
    sudo swapoff -a
    ```

    - comment out the swap drive from fstab

    ```bash
    # swap was on ...
    # UUID=XXXXXXX-XXXXX-XXXX
    ```


## [Repository of scripts](https://github.com/harryho/kube-cluster-in-5mins.git)

<file_sep>+++
title = "Sql Server Note - 1"
description="Introduction of MS Sql Server"
+++

## SQL Server

> Microsoft SQL Server is a relational database management system developed by Microsoft. As a database server, it is a software product with the primary function of storing and retrieving data as requested by other software applications—which may run either on the same computer or on another computer across a network (including the Internet). 




### Get db/table size or space

* Get db size

```sql
-- Get database size 
SELECT      sys.databases.name,  
            CONVERT(VARCHAR,SUM(size)*8/1024)+' MB' AS TotalDiskSpaceMB  ,
            CONVERT(VARCHAR,SUM(size)*8/1024/1024)+' GB' AS TotalDiskSpaceGB
FROM        sys.databases   
JOIN        sys.master_files  
ON          sys.databases.database_id=sys.master_files.database_id  

GROUP BY    sys.databases.name  
ORDER BY     TotalDiskSpaceMB

-- Get database space & unallocated space
exec sp_spaceused

-- 

```

* Get table size

```sql
SELECT  
        t.NAME AS TableName,
        s.Name AS SchemaName,
        p.rows AS RowCounts,
        SUM(a.total_pages) * 8 AS TotalSpaceKB,
        CAST(ROUND(((SUM(a.total_pages) * 8) / 1024.00), 2) AS NUMERIC(36, 2))
        AS TotalSpaceMB,
        SUM(a.used_pages) * 8 AS UsedSpaceKB, 
        CAST(ROUND(((SUM(a.used_pages) * 8) / 1024.00), 2) AS NUMERIC(36, 2))
        AS UsedSpaceMB,
        (SUM(a.total_pages) - SUM(a.used_pages)) * 8 AS UnusedSpaceKB,
        CAST(ROUND(((SUM(a.total_pages) - SUM(a.used_pages)) * 8) / 1024.00, 2) AS NUMERIC(36, 2))
        AS UnusedSpaceMB

FROM sys.tables t
INNER JOIN sys.indexes i ON t.OBJECT_ID = i.object_id
INNER JOIN sys.partitions p ON i.object_id = p.OBJECT_ID AND i.index_id = p.index_id
INNER JOIN sys.allocation_units a ON p.partition_id = a.container_id
LEFT OUTER JOIN sys.schemas s ON t.schema_id = s.schema_id

WHERE t.NAME NOT LIKE 'dt%' 
  AND t.is_ms_shipped = 0
  AND i.OBJECT_ID > 255 
GROUP BY t.Name, s.Name, p.Rows
ORDER BY t.Name;

```



### Get full text search objects

```sql
SELECT 
    SCHEMA_NAME(tbl.schema_id) as SchemaName,
    tbl.name AS TableName, 
    FT_ctlg.name AS FullTextCatalogName,
    i.name AS UniqueIndexName,
    scols.name AS IndexedColumnName
FROM 
    sys.tables tbl
INNER JOIN 
    sys.fulltext_indexes FT_idx 
ON 
    tbl.[object_id] = FT_idx.[object_id] 
INNER JOIN 
    sys.fulltext_index_columns FT_idx_cols
ON 
    FT_idx_cols.[object_id] = tbl.[object_id]
INNER JOIN
    sys.columns scols
ON 
    FT_idx_cols.column_id = scols.column_id
    AND FT_idx_cols.[object_id] = scols.[object_id]
INNER JOIN 
    sys.fulltext_catalogs FT_ctlg
ON 
    FT_idx.fulltext_catalog_id = FT_ctlg.fulltext_catalog_id
INNER JOIN 
    sys.indexes i
ON 
        FT_idx.unique_index_id = i.index_id
    AND FT_idx.[object_id] = i.[object_id];

```

### Find the table

* Find table by naming pattern

```sql 
SELECT     distinct	t.name AS 'TableName'
FROM        sys.columns c
JOIN        sys.tables  t   ON c.object_id = t.object_id
WHERE       t.name LIKE '%bk%'
ORDER BY    TableName;
```

* Find table by colume name


```sql
SELECT      c.name  AS 'ColumnName'
            ,t.name AS 'TableName'
FROM        sys.columns c
JOIN        sys.tables  t   ON c.object_id = t.object_id
WHERE       c.name LIKE '%MyName%'
ORDER BY    TableName
            ,ColumnName;
```



### Restore user login after db restore


```sql
EXEC sp_change_users_login Report

EXEC sp_change_users_login 'Auto_Fix', 'your_username', NULL, 'your_password';
```


### Create a new login 


```sql
USE [master]
GO
CREATE LOGIN [sql_user_id] WITH PASSWORD=N'<PASSWORD>', 
DEFAULT_DATABASE=[Your_Database], CHECK_EXPIRATION=OFF, CHECK_POLICY=OFF
GO

USE [Your_Database]
GO
CREATE USER [sql_user_id] FOR LOGIN [sql_user_id]
GO
ALTER USER [sql_user_id] WITH DEFAULT_SCHEMA=[sql_user_id]
GO
CREATE SCHEMA [sql_user_id] AUTHORIZATION [sql_user_id]
GO
ALTER ROLE [db_datareader] ADD MEMBER [sql_user_id]
GO
ALTER ROLE [db_datawriter] ADD MEMBER [sql_user_id]
GO
```

### Get Connection Info

```sql

EXEC sp_who
GO

EXEC sp_who @loginname='user_id'
GO

-- Enhanced verssion
EXEC sp_who2
GO
```
<file_sep>+++

date = "2016-12-07T16:56:21+11:00"
title = "Laravel MVC Starter"

description = "This starter is the starting point of laravel 5 MVC project. "
+++


## Summary

This starter is the starting point of laravel 5 MVC project. This application is meant to be used as a starting place for those looking to get their feet wet with laravel. 


## Overview of project

### BDD ( Business domain design)

``` ini
+-------+ 0...*     0...* +--------+ 1     0...* +---------+ 
| tag   | --------------- |  post  | ----- ----- | comment |
+-------+                 +--------+             +---------+           
                               | 1...*
                               |
                               | 0...*
                          +--------+ 
                          |  like  |
                          +--------+

```


### Structure of starter

```
\path\to\lara-mvc-starter
+---app                 <-// Customized PHP application code
|   +---Console
|   +---Exceptions
|   +---Http
|   |   +---Controllers
|   |   |   \---Auth
|   |   \---Middleware
|   \---Providers
+---bootstrap          <-// bootstrap the framework and configure autoloading
|   \---cache
+---config              <-// application configuration
+---database            <-// database migration files
|   +---factories
|   +---migrations
|   \---seeds
+---public               <-// Distributed website folder including Style sheet
+---resources            <-// View files, Javascripts, localization setting
|   +---assets
|   +---lang
|   \---views
|       +---admin
|       +---blog
|       +---errors
|       +---layouts
|       +---other
|       +---partials
|       \---vendor
|           \---pagination
+---routes              <-// Route definitions and setting
+---storage             <-//Blade templates, file based sessions, file caches
|   +---app
|   |   \---public
|   +---framework
|   |   +---cache
|   |   +---sessions
|   |   \---views
|   \---logs
+---tests
\---vendor            <-// Laravel framework

```


## Screenshots

> ![laravel-mvc-starter](/img/lara-mvc-starter.png)

### Browse [Repository](https://github.com/harryho/lara-mvc-starter.git)


<file_sep>+++

date = "2018-01-04T13:59:31+11:00"
title = "Lubuntu 16 desktop"
description = "Post-installation for Lubuntu 16 desktop"
draft = false
+++

Prelude

> There is no big difference against setup between Lubuntu and Ubuntu. I just want to keep a latest version of setup for myself as reference

## Prerequisites

* You have Lubuntu 16 in place
* Internet is available 

## UFW setup

```bash
sudo ufw enable
sudo ufw allow 80/tcp
sudo ufw allow ssh
sudo ufw allow 443/tcp
sudo ufw allow 8000/tcp
```

## SSH server setup

`!!! For production environment, SSH should be secured by the CA`

```bash
sudo apt-get install openssh-server

## backup default config
sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.factory-defaults
sudo chmod a-w /etc/ssh/sshd_config.factory-defaults

## use any editor to update sshd_config
sudo nano /etc/ssh/sshd_config

## uncomment  PasswordAuthentication yes to allow remote password login
## Password authentication is only for test environment

## setup ssh auto-start onboot
sudo update-rc.d ssh defaults
```

## !!! Install the software-properties-common Package

```bash
sudo apt-get install software-properties-common python-software-properties
```

## Time Zone setup

```bash
sudo dpkg-reconfigure tzdata
```

## Install tmux

```bash
sudo apt-get install tmux
```

* Most useful tmux commands

> Ctrl+b " — split pane horizontally.
>
> Ctrl+b % — split pane vertically.
>
> Ctrl+b arrow key — switch pane.
>
> Hold Ctrl+b, don’t release it and hold one of the arrow keys — resize pane.
>
> Ctrl+b c — (c)reate a new window.
>
> Ctrl+b , — rename reate a new window.
>
> Ctrl+b n — move to the (n)ext window.
>
> Ctrl+b p — move to the (p)revious window.

## Install git

```bash
sudo add-apt-repository ppa:git-core/ppa
sudo apt-get update
sudo apt-get install git
```

## install docker CE (Ubuntu 16 LTS)

```bash
## Update the apt package index
sudo apt-get update

## Install packages to allow apt to use a repository over HTTPS
sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common

## Add Docker’s official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -



## Verify the last 8 characters of the fingerprint.
sudo apt-key fingerprint xxxxxxxx


##   set up the stable repository
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"

## apt update
sudo apt-get update

## install docker CE
sudo apt-get install docker-ce
```

## Install JDK 9

* Downlaod the JDK from Oracle website.

```bash
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java9-installer

java -version
```

* Setup environment

```bash
sudo apt-get install oracle-java9-set-default
sudo apt autoremove

## Following setup is no longer required
## sudo su
## cat >> /etc/environment <<EOL
## JAVA_HOME=/usr/lib/jvm/java-9-oracle
## JRE_HOME=/usr/lib/jvm/java-9-oracle/jre
## EOL
```

* Test JDK with a simple HelloWorld program

```java
import java.util.Calendar;

class HelloWorld {
    public static void main(String[] args) {
        Calendar cal = Calendar.getInstance();
        int year = cal.get(Calendar.YEAR);
        int month = cal.get(Calendar.MONTH) + 1;
        int day = cal.get(Calendar.DATE);
        int hour = cal.get(Calendar.HOUR_OF_DAY);
        int minute = cal.get(Calendar.MINUTE);
        String username = System.getProperty("user.name");
        System.out.println(username+ ": Hello World! ");
        System.out.println(year + "/" + month + "/" + day + " " + hour + ":" + minute);
    }
}
```

* Compile and run the program

```bash
javac HelloWorld.java
java HelloWorld.java
```

## Install nodejs

* Install Nodejs 8.x

```bash
curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs
```

* Install latest npm, yarn and ts

```
sudo npm install -g npm
sudo npm install -g typescript
sudo mpm install -g yarn
```

## Install PHP

* Add new repo

```bash
sudo apt-get install -y python-software-properties
sudo add-apt-repository -y ppa:ondrej/php
sudo apt-get update -y

apt-cache pkgnames | grep php7.2
```

* Option 1: Install LAMP stack

```bash
sudo apt-get install -y apache2
sudo apt-get install -y php7.2 libapache2-mod-php7.2 \
    php7.2-cli php7.2-common php7.2-mbstring php7.2-gd \
    php7.2-intl php7.2-xml php7.2-mysql php7.1-mcrypt php7.2-zip
```

* Option 2: Install LEMP stack

```bash
sudo apt-get install -y nginx
sudo apt-get install -y php7.2 php7.2-fpm php7.2-cli \
    php7.2-common php7.2-mbstring php7.2-gd php7.2-intl \
    php7.2-xml php7.2-mysql php7.1-mcrypt php7.2-zip
```

* Disable Apache and Nginx if you install both

```bash
sudo systemctl disable apache2.service
sudo systemctl disable nginx.service
```

## Install Python2, Python3

* Ubuntu has python2 installed by default

```bash
sudo apt-get python-pip
sudo apt-get install python3-pip
sudo apt-get install python3-dev python-dev

## Install virtualenv
sudo pip install virtualenv
sudo pip3 install virtualenv
```

* Install new python 3.6

```
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt-get update
sudo apt-get install python3.6
```

## Install Go

* Install Go

```bash
wget https://dl.google.com/go/go1.10.1.linux-amd64.tar.gz


## check hash
shasum -a 256 go*linux-amd64.tar.gz

## install tar ball
sudo tar -xvzf go*linux-amd64.tar.gz
sudo mv go /usr/local
```

* Setup GOROOT & GOPATH

```bash
export GOROOT="/usr/local/go"
export GOPATH="$HOME/ws/go"
export PATH="$GOROOT/bin:$GOPATH/bin:$PATH"
```

* Create a simple `hello.go` file to test

  `touch ~/ws/go/src/hello/hello.go`

```go
package main

import (
    "fmt"
    "log"
    "os/user"
)

func main(){
    user, err := user.Current()
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf(user.Name + " said : Hello World! \n" )
}
```

* Run the program

```bash
go run $GOPATH/src/hello/hello.go
go install hello
$GOPATH/bin/hello
```

* Install hugo

```
# use apt
sudo apt install hugo

# use snap
sudo snap install hugo
```

## Install clang & cmake

```bash
sudo apt-get install clang
sudo apt-get install cmake
```

## Install Rust

```bash
$ curl -f -L https://static.rust-lang.org/rustup.sh -O
$ sh rustup.sh
```

## Install vim 8

* Add ppa repo

```
sudo add-apt-repository ppa:jonathonf/vim
sudo apt update
sudo apt install vim
```

* Install awesome vimrc

```
git clone --depth=1 https://github.com/amix/vimrc.git ~/.vim_runtime
sh ~/.vim_runtime/install_awesome_vimrc.sh
```

## Install MySql

* Install mysql 

```
wget https://dev.mysql.com/get/mysql-apt-config_0.8.9-1_all.deb
sudo dpkg -i mysql-apt-config_0.8.9-1_all.deb
sudo apt-get install mysql-server
systemctl status mysql
mysqladmin -u root -p version

mysql -u root -p mysql
```

* create a sample table products

```
CREATE TABLE products (
    id INT AUTO_INCREMENT NOT NULL,
    title VARCHAR(255),
    price DECIMAL(10, 2)  NOT NULL,
    created_at datetime,
    deleted_at datetime,
    tags VARCHAR(255)
    ,PRIMARY KEY (id)
);

load data local infile '/home/<your_name>/db/products.csv' into table products \
fields terminated by ',' enclosed by '"' lines terminated by '\n' \
(id, title, price, created_at, deleted_at, tags);

```

## Install PostgresQL

* psql is case sensitive

```
echo 'deb http://apt.postgresql.org/pub/repos/apt/ xenial-pgdg main' \
    >> /etc/apt/sources.list.d/pgdg.list
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | \
    sudo apt-key add -

sudo apt-get update
sudo apt-get install postgresql-10

sudo su - postgres
psql -U postgres

## Create a dump databbase
curl -L -O http://cl.ly/173L141n3402/download/example.dump
createdb pgguide
pg_restore --no-owner --dbname pgguide example.dump
psql --dbname pgguide

psql 

## Rename database -- use double quote 
ALTER database "pgguide" rename to "sample"
```

* export the database to sql file

```
sudo su postgres
pg_dump sample >> sample.sql
```

* export table to csv file

```
COPY products to '/home/<your_name>/db/products.csv' delimiter ',' csv; 
```

* export data to json file

```
select json_agg(t) from \
    (select * from products) t \t on \pset format unaligned \g products.json
```



## Install mongodb

```
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 \ 
    --recv 2930ADAE8CAF5059EE73BB4B58712A2291FA4AD5
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu \
  xenial/mongodb-org/3.6 multiverse" \
  | sudo tee /etc/apt/sources.list.d/mongodb-org-3.6.list

sudo apt-get update
sudo apt-get install -y mongodb-org

sudo service mongod start
sudo service mongod stop
sudo service mongod status

```

* Create a database `sample` and insert one record into document `products`

```json
use sample
db.products.insertOne({
    id: 1,
    title: "Dictionary", 
    price: 9.99, 
    created_at: "2011-01-02 07:00:00+11", 
    tags: "{Book}"
});

db.products.find();
```

* Import json into database


```
mongoimport --db sample --collection products --drop \
    --jsonArray --file ~/db/products.json
```<file_sep>+++
title = "Every IT project involves culture change"
description="Culture Change"
draft = true
+++

## Every IT project involves culture change

    <file_sep>+++
title="Error handling"
description="Error handling"
weight=5
+++


### Error handling

* Go does not have an exception mechanism, like the try/catch in Java or .NET for instance: you cannot throw exceptions. Instead it has a defer-panic-and-recover mechanism.

* The Go way to handle errors is for functions and methods to return an error object as their only or last return value—or nil if no error occurred—and for calling functions to always check the error they receive.

* Handle the errors and return from the function in which the error occurred with an error message to the user: that way if something does go wrong, your program will continue to function and the user will be notified. The purpose of panic-and-recover is to deal with genuinely exceptional (so unexpected) problems and not with normal errors.

* The idiomatic way in Go to detect and report error-conditions

    - A function which can result in an error returns two variables, a value and an error-code; the latter is nil in case of success, and != nil in case of an error-condition.
    - After the function call the error is checked, in case of an error ( if error != nil) the execution of the actual function (or if necessary the entire program) is stopped.


__!!Never ignore errors, because ignoring them can lead to program crashes!!__


#### Error interface

* Go has a predefined error interface type:

    ```go
    type error interface {
        Error() string
    }
    ```

* Defining errors

    ```go
    err := errors.New("math - square root of negative number")
    ```
* Making an error-object with fmt

    
    ```go
    if f < 0 {
        return 0, fmt.Errorf("math: square root of negative number %g", f)
    }
    ```

### Run-time exceptions & panicking

* When execution errors occur, such as attempting to index an array out of bounds or a type assertion failing, the Go runtime triggers a run-time panic with a value of the interface type runtime.Error, and the program crashes with a message of the error; this value has a RuntimeError()-method, to distinguish it from a normal error.

* A panic can also be initiated from code with the panic function is used, which effectively creates a run-time error that will stop the program. It takes 1 argument of any type, usually a string, to be printed out when the program dies. The Go runtime takes care to stop the program and issuing some debug information.

* If panic is called from a nested function, it immediately stops execution of the current function, all defer statements are guaranteed to execute and then control is given to the function caller, which receives this call to panic. This bubbles up to the top level, executing defers, and at the top of the stack the program crashes and the error condition is reported on the command-line using the value given to panic: this termination sequence is called __panicking__.



### Error-handling & panicking in custom package

* Best practice for custom package

    a. Always recover from panic in your package: no explicit panic() should be allowed to cross a package boundary

    b. Return errors as error values to the callers of your package.


* Sample

    * Parse package

        ```go
        package parse
        import (
            "fmt"
            "strings"
            "strconv"
        )
        // A ParseError indicates an error in converting a word into an integer.
        type ParseError struct {
            Index int // The index into the space-separated list of words.
            Word string // The word that generated the parse error.
            // The raw error that precipitated this error, if any.
            Error err
        }
        // String returns a human-readable error message.
        func (e *ParseError) String() string {
            return fmt.Sprintf("pkg parse: error parsing %q as int", e.Word)
        }
        // Parse parses the space-separated words in in put as integers.
        func Parse(input string) (numbers []int, err error) {
            defer func() {
                if r := recover(); r != nil {
                    var ok bool
                    err, ok = r.(error)
                    if !ok {
                    err = fmt.Errorf("pkg: %v", r)
                    }
                }
            }()
            fields := strings.Fields(input)
            numbers = fields2numbers(fields) // here panic can occur
            return
        }
            
        func fields2numbers(fields []string) (numbers []int) {
            if len(fields) == 0 {
                panic("no words to parse")
            }
            for idx, field := range fields {
                num, err := strconv.Atoi(field)
                if err != nil {
                panic(&ParseError{idx, field, err})
                }
            numbers = append(numbers, num)
        }
        return
        }
        ```
    * main package

        ```go
        func main() {
            var examples = []string{
                "1 2 3 4 5",
                "100 50 25 12.5 6.25",
                "2 + 2 = 4",
                "1st class",
                ""
            }

            for _, ex := range examples {
                fmt.Printf("Parsing %q:\n ", ex)
                nums, err := parse.Parse(ex)
                if err != nil {
                    // here String() method from ParseError is used
                    fmt.Println(err)
                continue
            }

            fmt.Println(nums)
        }

        /* Output:
            Parsing "w1 2 3 4 5":
            360
            <NAME>
            [1 2 3 4 5]
            Parsing "100 50 25 12.5 6.25":
            pkg parse: error parsing "12.5" as int
            Parsing "2 + 2 = 4":
            pkg parse: error parsing "+" as int
            Parsing "1st class":
            pkg parse: error parsing "1st" as int
            Parsing "":
            pkg: no words to parse
        */

        ```


### Recover

* recover is only useful when called inside a deferred function (see § 6.4) : it then retrieves the error value passed through the call of panic; when used in normal execution a call to recover will return nil and have no other effect.

* Summarized: panic causes the stack to unwind until a deferred recover() is found or the program terminates 


#### Similar try-catch block in Go


```go
func protect(g func()) {
    defer func() {
        log.Println("done")
        // Println executes normally even if there is a panic
        if err := recover(); err != nil {
          log.Printf("run time panic: %v", err)
        }
    }()
    log.Println("start")
    g() // possible runtime-error
}
```

#### Sample of panic, defer & recover

```go
func badCall() {
    panic("bad end")
}

func test() {
    defer func() {
        if e := recover(); e != nil {
            fmt.Printf("Panicking %s\r\n", e)
        }
    }()
    badCall()
    fmt.Printf("After bad call\r\n")
}

func main() {
    fmt.Printf("Calling test\r\n")
    test()
    fmt.Printf("Test completed\r\n")
}
```

### An error-handling scheme with closures

* Combining the defer/panic/recover mechanism with closures can result in a far more elegant scheme that we will now discuss. However it is only applicable when all functions have the same signature, which is rather restrictive. 


* The scheme uses 2 helper functions:

    i) check: a function which tests whether an error occurred, and panics if so:

    ```go
    func check(err error) { if err != nil { panic(err) } }
    ```

    ii) errorhandler: this is a wrapper function. It takes a function fn of our type fType1 and returns such a function by calling fn. However it contains the defer/recover mechanism

    ```go
    func errorHandler(fn fType1) fType1 {
        return func(a type1, b type2) {
            defer func() {
            if e, ok := recover().(error); ok {
               log.Printf("run time panic: %v", err)

            }
            }()
            fn(a, b)
        }
    }
    ```

### Start external program

```go
func main() {
    // 1) os.StartProcess //
    /*********************/
    /* Linux: */
    env := os.Environ()
    procAttr := &os.ProcAttr{
        Env: env,
        Files: []*os.File{
            os.Stdin,
            os.Stdout,
            os.Stderr,
        },
    }
    pid, err := os.StartProcess("/bin/ls", []string{"ls", "-l"}, procAttr)
    if err != nil {
        fmt.Printf("Error %v starting process!", err) //
        os.Exit(1)
    }
    fmt.Printf("The process id is %v", pid)
    /* Output:
    The process id is &{21275 0 0 {{0 0} 0 0 0 0}}The process id is &{21276 0 0 {{0 0} 0 0 0 0}}total 54
    -rwxrwxrwx 1 root root   250 Sep 21 19:33 csv_data.txt
    -rwxrwxrwx 1 root root 25227 Oct  4 23:34 hello.go
    -rwxrwxrwx 1 root root  6708 Sep 21 10:25 hello.go.txt
    -rwxrwxrwx 1 root root   130 Sep 21 11:08 output.txt
    -rwxrwxrwx 1 root root  8898 Sep 21 12:10 target_hello.txt
    -rwxrwxrwx 1 root root  1619 Sep 22 14:40 urlshorten.go.txt
    -rwxrwxrwx 1 root root   182 Sep 21 13:50 vcard.json
    */
    // 2nd example: show all processes
    pid, err = os.StartProcess("/bin/ps", []string{"-e", "opid,ppid,comm"}, procAttr)
    if err != nil {
        fmt.Printf("Error %v starting process!", err) //
        os.Exit(1)
    }
    fmt.Printf("The process id is %v", pid)
    // 2) cmd.Run //
    /***************/
    cmd := exec.Command("gedit") // this opens a gedit-window
    err = cmd.Run()
    if err != nil {
        fmt.Printf("Error %v executing command!", err)
        os.Exit(1)
    }
    fmt.Printf("The command is %v", cmd)
}
```

### Testing

* Table-driven test


    ```go
    var tests = [] struct {
        in
        // Test table
        string
        out string
    }{
        {"in1", "exp1"},
        {"in2", "exp2"},
        {"in3", "exp3"},
    // ....
    }

    func verify(t *testing.T, testnum int, testcase, input, output, expected string) {
        if input != output {
            t.Errorf("%d. %s with input = %s: output %s != %s", testnum, testcase, input, output, expected)
        }
    }

    func TestFunction(t *testing.T) {
        for i, tt := range tests {
            s := FuncToBeTested(tt.in)
            verify(t, i, "FuncToBeTested: ", tt.in, s, tt.out)
        }
    }
    ```<file_sep>+++

title = "Package & Module"
description="Good practices for package & module"
weight=1
+++

## Package & Module
---

* Packages are modules that contain other modules.
* Packages are generally implemented as directories containing a special `__init__.py` file.
* The `__init__.py` file is executed when the package is imported.
* Packages can contain sub packages which themselves are implemented with `__init__.py` files in directories.
* The module objects for packages have a `__path__` attribute.

### sys.path

* List of directories which Python searches for modules.


    ```python
    # list directories
    >>>import sys
    >>>sys.path
    ```


* Use `append` to attach the package directory to sys.path 
    * Append the package to sys.path
    * If you append the relative path of the package to sys.path, you need to make sure the it is correct. 

* Example

    * path: path_root\package0\module0.py
    * The code of module0.py

        ```python
        def test():
            print('module0 -- test !')
        ```
    
    * Test module importing


        ```python
        cd root
        python
        >>>import sys
        >>>sys.append('package0')
        >>>import module0
        >>>module0.test
        module0 -- test !
        >>>exit()

        # It will fail if you launch python at the parent directory of root 

        cd ..
        python
        >>>import sys
        >>>sys.append('package0')
        >>>import module0
        Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
        ImportError: No module named 'module0'

        ### It will success if you adjust relative path as below
        >>>sys.append('path_root/package0')
        >>>import module0
        >>>module0.test
        module0 -- test !
        ```

## PYTHONPATH

* Environment variable adds paths to sys.path
* Use previous `module0.py` to test
* Linux 

    ```bash
    export PYTHONPATH=package0
    python
    >>>import module0
    module0 -- test !
    ```

* Windows

    ```
    set PYTHONPATH=package0
    python
    >>>import module0
    module0 -- test !
    ```

### Package structure
---


* Convert a package into a module
* Basic structure of package0

```bash
path_root  <--// it must be attached sys.path
+---package0    <--// package root 
    +---__init__.py  <--// package init file
    \---module0.py
```

* Sample code -  `__init__.py` ( The sample code is for demo purpose)

```python
print('package0 --init...')
```

* Test 

```
python
>>>import package0
package0 --init...
>>>import package0.module0
>>>package0.module0.test()
module0 -- test !
```

* Add a `FileReader` class into `module0.py`
* Sample code of `module0.py`

    ```python
    class FileReader:
        def __init__(self, filename):
            self.filename = filename
            self.f = open(self.filename, 'rt')

        def close(self):
            self.f.close()
        
        def read(self):
            return self.f.read()

    ```

* Test

```python
fr=package0.module0.FileReader('package0/module0.py')
>>>fr.read()
>>>fr.close()

```

* Update `__init__.py`

```
from package0.module0 import FileReader
```

* Test again

```python
>>>import package0
>>>r=package0.FileReader('package0/module0.py')
>>>r.read()
>>>r.close()
```


### Subpackage
---

* Demo below shows how to add subpackages
* Add sub-package under the `package0`
* Structure

```
path_root  <--// it must be attached sys.path
+---package0    <--// package root 
    +---compress
    |   +---__init__.py
    |   +---bz.py
    |   +---gz.py
    +---__init__.py  <--// package init file
    \---module0.py
```

* Sample code - `gz.py`

```python
import gzip
import sys

opener=gzip.open

if __name__ == '__main__':
    f = gzip.open(sys.argv[1], mode='wt')
    f.write(' '.join(sys.argv[2:]))
    f.close()
```

* Sample code - `bz.py`

```python
import bz2
import sys

opener = bz2.open

if __name__ == '__main__':
    f = bz2.open(sys.argv[1], mode='wt')
    f.write(' '.join(sys.argv[2:]))
    f.close()
```

* Test by creating two compressed files 

```
python3 -m package0.compress.gz test.gz data compressed with gz
python3 -m package0.compress.bz test.bz2 data compressed with bz2
```

* Change `FileReader.py` to read above files

```python
from package0.compress import gz, bz
import os

extension_map = {
    '.gz':gz.opener,
    '.bz2':bz.opener
}

class FileReader:
    def __init__(self, filename):
        self.filename = filename
        extension = os.path.splitext(filename)[1]
        opener = extension_map.get(extension, open)
        self.f = opener(self.filename, 'rt')

    def close(self):
        self.f.close()
    
    def read(self):
        return self.f.read()

```


* Test

```python
>>> import package0
>>> r=package0.FileReader('test.gz')
>>> r.read()
'data compressed with gz'
>>> r=package0.FileReader('test.bz2')
>>> r.read()
'data compressed with bz2'

```


### Import with relative path
---

* Example below show how to use relative path to import packages 

```
path_root 
+---package0     
    +---compress
    |   +---__init__.py
    |   +---bz.py      <--// from ..module0 import FileReader
    |   +---gz.py      <--// from .bz import bz.opener
    +---__init__.py  
    \---module0.py

```

### Namespace package
---

* Namespace packages have no `__init__.py`
* Python scans all entries in `sys.path`.
* If a matching directory with `__init__.py` is found, a normal package is loaded
* Otherwise, all matching directories in `sys.path` are considered part of the namespace package
* Example 1

    * Structure of package

    ```
    path_root0 
    +---package0
        +---module0.py
    path_root1 
    +---package0
        +---module0.py    
    ```

    * Test

    ```
    >>> import sys
    >>> sys.path.append('gh')
    >>> sys.path.append('path_root0')
    >>> sys.path.append('path_root1')
    >>> import package0
    >>> package0.__path__
    _NamespacePath(['gh\\package0', 'path_root0\\package0', 'path_root0\\package0'])
    ```

* Example 2

    * Structure of package

    ```
    path_root0 
    +---package0
        +---module0.py
    path_root1 
    +---package0
        +---module0.py    
    path_root2 
    +---package0
        +---__init__.py  <--// Namespace should not include __init__.py
        +---module0.py            
    ```
    
    * Test

    ```
    >>> import sys
    >>> sys.path.append('path_root0')
    >>> sys.path.append('path_root1')
    >>> sys.path.append('path_root2')
    >>> import package0
    >>> package0.__path__
    _NamespacePath(['path_root2\\package0'])    
    ```



### Recommended Executable directories
---

```
project_root <--// Project root directory contains everything
+---__main__.py
+---project_name
|   +---__init__.py
|   +---resource.py
|   +---package0
|   |   +---__init__.py
|   |   +---module0.py
|   +---test
|      +---__init__.py
|      +---test.py
\---setup.py      

```

### Visual Studio Code setup
---

* Install python plugin `Python - donjayamanne`
* Setup `launch.json`

```json
{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python",
            "type": "python",
            "request": "launch",
            "stopOnEntry": true,
            "pythonPath": "${config:python.pythonPath}",
            "program": "${workspaceRoot}/__main__.py",
            "cwd": "${workspaceRoot}",
            "env":{},
            "envFile": "${workspaceRoot}/.env",
            "debugOptions": [
                "WaitOnAbnormalExit",
                "WaitOnNormalExit",
                "RedirectOutput"
            ]
        }
    ]
    ...
}
```


* Sample code - `__main__.py`

    ```python
    from package1 import FileReader

    if __name__ == "__main__":
        app = FileReader('C:/ws/python/plural/pbb/gh/test.gz')
        print(app.read())
        app.close()

        app = FileReader('C:/ws/python/plural/pbb/gh/test.bz2')
        print(app.read())
        app.close()
    ```



### Function & Lambda

## Function

* statement which defines a function and binds it to a name
* Must have a name
* Arguments delimited by parentheses, separated by commas
* Zero or more arguments supported -zero arguments ⇒ empty parentheses
* Body is an indented block of statements
* A return statement is required to return anything other than None
* Regular functions can have docstrings‣ Easy to access for testing

## Lambda

* Expression which evaluates to a function
* Anonymous
* Argument list terminated by colon, separated by commas
* Zero or more arguments supported - zero arguments ⇒ lambda:
* Body is a single expression
* The return value is given by the body expression. No return statement is permitted.
* Lambdas cannot have docstrings
* Awkward or impossible to test
* Example

```python
>>>names=list(['<NAME>', '<NAME>', '<NAME>'])
>>>sorted(names, key=lambda name:name.split()[-1]))
['<NAME>', '<NAME>', '<NAME>']
```

## Callable
---
* Callable instance

    * Example 
    * Sample code - `Resolver.py`

    ```python
    class Resolver:

        def __init__(self):
            self.cache={}

        def __call__(self, host):
            if host not in self.cache:
                self.cache[host]= socket.gethostbyname(host)
            return self.cache[host]
    ```

    * Sample code - `__main__.py`

    ```
    from package1 import Resolver

    if __name__ == "__main__":
        app = Resolver()
        print(app('harryho.github.io'))
        print(app.__call__('harryho.github.io'))
    ```

* Callable class

```python
>>>seq_class_1 = list
>>>sequence= seq_class_1('abc')
>>>type(sequence)
<class 'list'>
>>>seq_class_1 = tuple
>>>sequence= seq_class_1('abc')
>>>type(sequence)
<class 'tuple'>
```



## Extended arguments and call
---

* Extended arguments - syntax:  `def extend( *args, **kargs)`

* Example

```python
>>> def tag(name, **attrs):
...     t='<'
...     for k,v in attrs.items():
...             t+= '{key}="{val}"'.format(key=k, val=str(v))
...     t+='>'
...     return t
...
>>> tag('a', href="harryho.github.io", target="_blank", id="link")
'<href="harryho.github.io"id="link"target="_blank">'
```


* Extended call - sample

```python
>>> def f1 ( a1, a2, *a3):
...     print(a1)
...     print(a2)
...     print(a3)
...
>>> aa=(2,3)
>>> f1(aa)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: f1() missing 1 required positional argument: 'a2'
>>> f1(*aa)
2
3
()

>>> def f2(a1, a2):
...     print(a1)
...     print(a2)
...

>>> f2(*aa)
2
3
>>> aa=(1,2,3,4)
>>> f2(*aa)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: f2() takes 2 positional arguments but 4 were given
```

* Transpose example 

```python
>>> mon=[12,13,15,12,14,18,13]
>>> tue=[11,14,16,12,11,17,14]
>>> for d in zip(mon, tue):
...     print(d)
...
(12, 11)
(13, 14)
(15, 16)
(12, 12)
(14, 11)
(18, 17)
(13, 14)
>>> daily = [mon, tue]
>>> from pprint import pprint as pp
>>> pp(daily)
[[12, 13, 15, 12, 14, 18, 13], [11, 14, 16, 12, 11, 17, 14]]
>>> transposed = list(zip(*daily))
>>> pp(transposed)
[(12, 11), (13, 14), (15, 16), (12, 12), (14, 11), (18, 17), (13, 14)]

```
<file_sep>+++


date = "2017-09-20T14:59:31+11:00"
title = "Grub Trouble Shooting"
description = "Common steps to trouble shoot the Grub booting problem "

draft = false
+++

## Update Grub Menu for dual OS boot

**Change BIOS**
* Start your PC by pressing a pressing a special function key (usually F12, F10 or F2 depending on the vendor specifications).
* Some PC's BIOS has `BOOT` tab option, open the `BOOT` tab, you will find the `OS Boot Manager`. It is the simplest way to fix the issue. If your PC's BIOS has no such setting feature, you need to check the next section. 

**Change the Windows Boot Manager**
* Login windows with common prompt 
* Restart windows, meanwhile press shift key
* In the options page, choose change to other options
* Troubleshooting
* Command Prompt
* Login in Windows with Common prompt
* Use BCDEdit to change windows boot manager. Change to boot ubuntu at first

```bash
REM backup
bcdedit /enum > X:\Users\public\documents\bcdedit.txt
REM change the bootmgr 
bcdedit /set {bootmgr} path \EFI\ubuntu\grubx64.efi
```
* After you reboot system, you will see the Grub 2 menu as follow.

```bash
                             GNU GRUB version 2.0
---------------------------------------------------------------------------------- 
| Ubuntu 
| Advanced options for Unbuntu
| Windows Boot Manager ( on /dev/sda2 )
| Fedora 20
| Advanced options for Fedora 20
| OpenSuse 
| Advanced options for OpenSuse
| ....

```


## Boot from Grub command promp

* Use <ESC> to navigate to grub command promp from Grub menu

* List all available dirves by typing `ls`. After that, you willl see a couple drives. If you have multiple hard drive, USB or SD Card pluged in. 

```
(hd0) (hd0,gpt4) (hd0, gpt3) (hd0,gpt2) (hd0, gpt1)  (hd1) (hd1,msdos2)(hd1, msdos2)(hd2)

## Get more detail of drives
ls -l
```

* From the above detail information, you might find the hard drive of your PC. Continue to use `ls` to locate the actual boot file to confirm the drive contains the boot file. 

```
## Assume the (hd0,gpt2) contains the linux kernal boot file. 
ls -a (hd0,gpt2)/
```

* Set root drive

```
set root=(hd0,gpt2)
linux (hd0,gpt2)/boot/vmlinuz-linux-4.4.x-xxx-generic
initrd (hd0,gpt2)/boot/initrd.img-linux-4.4.x-xxx-generic
normal
```




<file_sep>+++
title = "AWS: RDS - 1"
description = "RDS - Native Backup & Restore"
weight=20
+++


## RDS

Amazon Relational Database Service (Amazon RDS) makes it easy to set up, operate, and scale a relational database in the cloud. It provides cost-efficient and resizable capacity while automating time-consuming administration tasks such as hardware provisioning, database setup, patching and backups.

### Backup & Restore SQL Server

#### Backup database to S3

* Assumption
  * DB name: sample_db
  * S3 bucket name: sql-server-backup

* Backup with built-in stored proc

        exec msdb.dbo.rds_backup_database 
        @source_db_name='sample_db', 
        @s3_arn_to_backup_to='arn:aws:s3:::sql-server-backup/sample_db_20191221.bak', 
        @overwrite_S3_backup_file=1;


* Track status

        exec msdb.dbo.rds_task_status @db_name='sample_db'


#### Restore DB into EC2 

* Restore with powershell

        #Copy-S3Object -BucketName 'nsw-prod-s3-sql-backups' \ 
        # -Key sample_db-20190531011003.bak \
        # -LocalFile L:\Backups\Automated\sample_db-20190531011003.bak

        Restore-SqlDatabase -ServerInstance 'localhost' -Database "sample_db" \
        -BackupFile "L:\Backups\Automated\sample_db-20190531011003.bak" \
        -ReplaceDatabase -KeepReplication -Verbose

#### Restore DB into RDS

* Create IAM role which can access the backup file in the S3 bucket. e.g. **RDS_RESTORE_S3_READONLY**

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::sql-server-backup"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObjectMetaData",
                "s3:GetObject",
                "s3:PutObject",
                "s3:ListMultipartUploadParts",
                "s3:AbortMultipartUpload"
            ],
            "Resource": [
                "arn:aws:s3:::sql-server-backup/*"
            ]
        }
    ]
}
```
  

* Create a new option group, or copy or modify an existing option group.

* Add the **SQLSERVER_BACKUP_RESTORE** option to the option group.

* Associate the IAM role with the option. The IAM role must have access to an S3 bucket to store the database backups. e.g. **RDS_RESTORE_S3_READONLY**

* Associate the option group with the DB instance.

* Make sure the RDS has enough space to restore the db backup.

* Restore the database

```sql
EXEC msdb.dbo.rds_restore_database 
@restore_db_name='sample_db',
@s3_arn_to_restore_from='arn:aws:s3:::sql-server-backup/sample_db_20191221.bak'
GO
```

* Check the progress 

```sql
EXEC msdb.dbo.rds_task_status 
@db_name='sample_db'
GO
```

#### Caveat 

* After you modify the storage size for a DB instance, the status of the DB instance is storage-optimization. The DB instance is fully operational after a storage modification.

* You can't make further storage modifications until **six (6) hours** after storage optimization has completed on the instance.

* You can't reduce the amount of storage for a DB instance after storage has been allocated.


### Rename database

* Rename db with built-in stored proc

```sql
EXEC rdsadmin.dbo.rds_modify_db_name N'<OldName>', N'<NewName>'
```

* Troubleshoot -The database could not be exclusively locked to perform the operation.

```sql
# Kill connection
EXEC sp_who 
GO

KILL <spid>
GO

EXEC sp_who 
GO
```





<file_sep>+++

title="Projects"
weight = 1
+++

{{%children style="card" description="true" sort="Weight" %}}<file_sep>+++
title = "Adv Bash - 4"
description = "File & Standard IO "
draft=true
+++


## 


<file_sep>+++

title = "Iteration"
description="Iterables & Iteration"
weight=4
+++

## Iterables & Iteration

### Comprehensions

* Comprehensions can process more than one input sequence
* Multiple input sequences in comprehensions work like nested for-loops
* Comprehensions can also have multiple if-clauses interspersed with the for-clauses
* Later clauses in a comprehension can reference variables bound in earlier clauses
* Comprehension can also appear in the result expression of a comprehension, resulting in nested sequences

#### Example 
    
```python
## Comprehensions    
>>> points
[(0, 0), (0, 1), (0, 2), (0, 3), (1, 0), (1, 1), (1, 2), (1, 3), (2, 0), (2, 1), (2, 2), (2, 3)]
>>> points_c=[(x,y) for x in range(3) for y in range(4) ]
>>> points_c
[(0, 0), (0, 1), (0, 2), (0, 3), (1, 0), (1, 1), (1, 2), (1, 3), (2, 0), (2, 1), (2, 2), (2, 3)]
>>> points_d=[(x,y)
...     for x in range(3)
...     if x > 0
...     for y in range(4)
...     if y > x ]
>>> points_d
[(1, 2), (1, 3), (2, 3)]
>>> points_e = [ [ y-1 for y in range(x)  ] for x in range(3)]
>>> points_e
[[], [-1], [-1, 0]]
```

### Functional-style tools

* Python provides a number of functional-style tools for working with iterators

#### Map

* `map()` calls a function for each element in its input sequences
* `map()` returns an iterable object, not a fully-evaluated collection
* `map()` results are lazily evaluated, meaning that you must access them to
force their calculation
* `map()` results are typically evaluated through the use of iteration constructs such as for-loops
* You must provide as many input sequences to `map()` as the callable argument has parameters
* `map()` takes one element from each input sequence for each output element
it produces
* `map()` stops producing output when its shortest input sequence is exhausted 
* `map()` can be used to implement the same behavior as comprehensions in some cases

##### Example

```python
>>> sizes =['small','medium', 'large']
>>> colors = ['lavender', 'teal', 'burnt orange']
>>> animals = ['koala', 'platypus', 'salamander']
>>> def combine( size, color, animal):
...     return '{} {} {}'.format(size, color, animal)
...
>>> list(map(combine, sizes, colors, animals ))
['small lavender koala', 'medium teal platypus', 'large burnt orange salamander']
>>> import itertools
>>> def combine2(quantity, size, color, animal):
...     return '{} - {} {} {}'.format(quantity, size, color, animal)
...
>>> list(map(combine2, itertools.count(), sizes, colors, animals ))
['0 - small lavender koala', '1 - medium teal platypus', '2 - large burnt orange salamander']
```

#### Filter

* `filter()` selects values from an input sequence which match a specified criteria
* `filter()` passes each element in its input sequence to the function argument
* `filter()` returns an iterable over the input elements for which the function argument is truthy
* Like `map()`, `filter()` produces its output lazily If you pass None as the first argument to `filter()`, it yields the input values which evaluate to True in a boolean context `reduce()` cumulatively applies a function to the elements of an input sequence

##### Example

```python
## filter
>>> negs = filter(lambda x: x<0 , [3, -1, 2,-4,0,9,-33])
>>> list(negs)
[-1, -4, -33]
>>> notnones = filter(None, [0,1, False, True, [], () , {}, (1,2,), [1,2], '', 'yes'])
>>> list(map(type, list(notnones)))
[<class 'int'>, <class 'bool'>, <class 'tuple'>, <class 'list'>, <class 'str'>]
```

#### Reduce

* `reduce()` calls the input function with two arguments: the accumulated result so far, and the next element in the sequence
* `reduce()` is a generalization of summation
* `reduce()` returns the accumulated result after all of the input has been processed
* If you pass an empty sequence to `reduce()` it will raise a TypeError
* `reduce()` accepts an optional initial value argument
* This initial value is conceptually added to the front of the input sequence
* The initial value is returned if the input sequence is empty
* The `map()` and `reduce()` functions in Python are related to the ideas in the map-reduce algorithm 

##### Example

```python
## reduce
>>> from functools import reduce
>>> import operator
>>> reduce(operator.add, [1,2,3,4,5])
15

### x is interim result , y is the next sequence value
>>> def mul(x, y):
...     print(' mul {} {} '.format(x,y))
...     return x * y
...
>>> reduce( mul, range(1, 5))
mul 1 2
mul 2 3
mul 6 4
24
```

### The next() function

* Python's next() function calls `__next__()` on its argument
* Iterators in Python must support the `__next__()` method
* `__next__()` should return the next item in the sequence, or raise `StopIteration` if it is exhausted
* Python's `iter()` function calls `__iter__()` on its argument
* Iterable objects in Python must support the `__iter__()` method
* `__iter__()` should return an iterator for the iterable object 
* Objects with a `__getitem__()` method that accepts consecutive integer indices starting at zero are also iterables
* Iterables implemented via `__getitem__()` must raise `IndexError` when they are exhausted

##### Example of Iterator

```python
class ExampleIterator:
def __init__(self, data):
    self.index = 0
    self.data = data

def __iter__(self):
    return self

def __next__(self):
    if self.index >= len(self.data):
        raise StopIteration()

    rslt = self.data[self.index]
    self.index += 1
    return rslt
```

##### Example of Iterable
    
```python
class ExampleIterable:
    def __init__(self):
        self.data = [1, 2, 3]

    def __iter__(self):
        return ExampleIterator(self.data)
```

##### Example of AnotherIterable

```python
class AnotherIterable:
    def __init__(self):
        self.data = [1, 2, 3]

    def __getitem__(self, idx):
        return self.data[idx]
```


### The iter() function

* The extended form of `iter()` accepts a zero-argument callable and a sentinel value
* Extended `iter()` repeatedly calls the callable argument until it returns the sentinel value
* The values produced by extended `iter()` are those returned from the callable

##### Example

```python
>>> ts = iter(datetime.datetime.now, None)
>>> next(ts)
datetime.datetime(2017, 7, 14, 14, 38, 10, 752761)
>>> next(ts)
datetime.datetime(2017, 7, 14, 14, 38, 13, 373613)
>>> next(ts)
datetime.datetime(2017, 7, 14, 14, 38, 14, 754588)

## Read file 
## Content of the file file.txt
## You are reading 
## the file
## you won't read 
## the 
## END
## but not see the END above

>>> with open('file.txt', 'rt') as f:
...     for line in iter(lambda: f.readline().strip(), 'END'):
...             print(line)
...
You are reading
the file
you won't read
the
```

* One use case for extended `iter()` is to iterate using simple functions
* Protocol conforming iterators must also be iterable

##### Example of Sensor

```python
import datetime
import itertools
import random
import time

class Sensor:
    def __iter__(self):
        return self

    def __next__(self):
        return random.random()

sensor = Sensor()
timestamps = iter(datetime.datetime.now, None)

for stamp, value in itertools.islice(zip(timestamps, sensor), 10):
    print(stamp, value)
    time.sleep(1)

```



<file_sep>#!/bin/bash

if [ ! -z themes/docdock/layouts ]; then
    echo 'docdock does not exist.'
    git submodule add --force https://github.com/harryho/hugo-theme-docdock.git themes/docdock 
    git submodule init .
    git submodule update .
fi

DRAFT=$1

if [ ${DRAFT} == 'd' ]; then 
    echo 'start with draft '
    hugo serve -t docdock -D --watch  --disableFastRender
else
    hugo serve -t docdock --watch  --disableFastRender --ignoreCache
fi




<file_sep>+++
title = "TypeScript Note - 2"
description="Generics, Advanced Types"
draft=true
+++


### Generics

A major part of software engineering is building components that not only have well-defined and consistent APIs, but are also reusable. Components that are capable of working on the data of today as well as the data of tomorrow will give you the most flexible capabilities for building up large software systems.

* Generics with function

        function identity<T>(arg: T): T {
            return arg;
        }
        function identity2<T>(arg: Array<T>): Array<T> {
            console.log(arg.length);  // Array has a .length, so no more error
            return arg;
        }

        let myIdentity: <T>(arg: T) => T = identity;
        let myIdentity2: <T>(arg: T) => T = identity2; // Error 

* Generics with types

        interface GenericIdentityFn<T> {
            (arg: T): T;
        }

        function identity<T>(arg: T): T {
            return arg;
        }

        let myIdentity: GenericIdentityFn<number> = identity;


### Adavenced Types

#### Intersection Types 

An intersection type combines multiple types into one. This allows you to add together existing types to get a single type that has all the features you need. 





<file_sep>+++
date = "2018-11-30T14:59:31+11:00"
title = "Dual Boot Windows 10 & Ubuntu 18"
description = "Steps to create a machine with dual-boot OS: Windows 10 and Ubuntu 18"

+++


### Dual boot Ubuntu 18 with Windows 10

> I have a couple Linux workstations, but all of them are old PC or laptop. Today I get a chance to test Ubuntu on a brand new laptop. Here I are going to write down all I did to create this dual boot laptop 

#### Caution

The laptop I worked on is Lenovo IdeaPad S model with UEFI firmware, but it doesn't mean all Lenovo laptops will work in the same way, not to mention other brand's laptop. So before you try anything, please backup all your data first. 

#### Prepare bootable USB with Ubuntu ISO

* Prepare a clean USB pen with minimum 4G volume
* Download the 18.04 LTS Desktop ISO from Ubuntu website. 
* Create a bootable USB with Rufus or any other USB build tool


#### Prepare Windows Machine for Dual-Boot

* Start the command prompt as Admin

    * Start Menu -> Command Prompt (Admin) in order to enter Windows Command Line.

* Change Window boot manager to restart the system in Safe Mode

    ```bat
    bcdedit /enum 
    
    bcdedit /set {default} safeboot minimal

    ```

* The first thing is to create a free space on the computer hard disk in case the system is installed on a single partition.


* Once in CLI, type diskmgmt.msc on prompt and the Disk Management utility should open. From here, right click on C: partition and select Shrink Volume in order to resize the partition.

    ```cmd
    C:\Windows\system32\>diskmgmt.msc
    ```

* On Shrink C: enter a value on space to shrink in MB (use at least 20000 MB depending on the C: partition size) and hit Shrink to start partition resize as illustrated below (the value of space shrink from below image is lower and only used for demonstration purposes).

Once the space has been resized you will see a new unallocated space on the hard drive. Leave it as default and reboot the computer in order to proceed with Ubuntu installation.

* Reboot the Windows to UEFI configuration

* Change the UEFI setting

    * Disable the **OS Optimized Defaults**
    * Disable the **Intel Platform Trust Technology**  (Optional)
    * Change the boot drive order 

* Plugin the USB pen and continue to boot the system

#### Install Ubuntu

* Install Ubuntu Desktop as usual
* Recommand to install minimal version. You always can install other applications later.


#### Rollback the Windows Safe Mode

* Launch command promt as Admin and run following command

```
bcdedit /deletevalue {default} safeboot
```

#### Make your life easier

- Ubuntu

    * Update fstab to add the Windows mount point at system start up
    * Update the Grub file if you want to boot Windows as default OS

- Windows

    * Install the Linx Reader to access the Ubuntu files from Windows
    
     <file_sep>+++
date = "2011-03-09T10:59:31+11:00"
title = "COBIT 5"
description="COBIT 5 - Introduction & Principles"
weight=1
+++

## COBIT 5

COBIT (Control Objectives for Information and Related Technologies) is a business framework for the governance and management of Enterprise IT, created by ISACA.

The framework contains the COBIT 5 framework for governing and managing enterprise IT. It defines a set of generic processes for the management of IT, with each process defined together with process inputs and outputs, key process-activities, process objectives, performance measures and an elementary maturity model.


### Product Family

The COBIT 5 framework is built on five basic principles, which are covered in detail, and includes extensive guidance on enablers for governance and management of enterprise IT.

The COBIT 5 product family includes the following products:

* COBIT 5 (the framework)
* COBIT 5 enabler guides, in which governance and management enablers are discussed in detail. These include:
  * COBIT 5: Enabling Processes
  * COBIT 5: Enabling Information (in development)
  * Other enabler guides (check www.isaca.org/cobit)
* COBIT 5 professional guides, which include:
  * COBIT 5 Implementation
  * COBIT 5 for Information Security (in development)
  * COBIT 5 for Assurance (in development)
  * COBIT 5 for Risk (in development)
  * Other professional guides (check www.isaca.org/cobit)
* A collaborative online environment, which will be available to support the use of COBIT 5

COBIT 5 provides a comprehensive framework that assists enterprises in achieving their objectives for the governance and management of enterprise IT. Simply stated, it helps enterprises create optimal value from IT by maintaining a balance between realising benefits and optimising risk levels and resource use. COBIT 5 enables IT to be governed and managed in a holistic manner for the entire enterprise, taking in the full end-to-end business and IT functional areas of responsibility, considering the IT-related interests of internal and external stakeholders. COBIT 5 is generic and useful for enterprises of all sizes, whether commercial, not-for-profit or in the public sector.


### Principles

#### Principle 1: Meeting Stakeholder Needs

Enterprises exist to create value for their stakeholders by maintaining a balance between the realisation of benefits and the optimisation of risk and use of resources. COBIT 5 provides all of the required processes and other enablers to support business value creation through the use of IT. Because every enterprise has different objectives, an enterprise can customise COBIT 5 to suit its own context through the goals cascade, translating high-level enterprise goals into manageable, specific, IT-related goals and mapping these to specific processes
and practices.


#### Principle 2: Covering the Enterprise End-to-end

COBIT 5 integrates governance of enterprise IT into enterprise governance:

* It covers all functions and processes within the enterprise; COBIT 5 does not focus only on the ‘IT function’, but treats information and related technologies as assets that need to be dealt with just like any other asset by everyone in the enterprise.
* It considers all IT-related governance and management enablers to be enterprisewide and end-to-end, i.e., inclusive of everything and everyone—internal and external—that is relevant to governance and management of enterprise information and related IT.


#### Principle 3: Applying a Single, Integrated Framework

There are many IT-related standards and good practices, each providing guidance on a subset of IT activities. COBIT 5 aligns with other relevant standards and frameworks at a high level, and thus can serve as the overarching framework for governance and management of enterprise IT.


#### Principle 4: Enabling a Holistic Approach

Efficient and effective governance and management of enterprise IT require a holistic approach, taking into account several interacting components. COBIT 5 defines a set of enablers to support the implementation of a comprehensive governance and management system for enterprise IT. Enablers are broadly defined as anything that can help to achieve the objectives of the enterprise. The COBIT 5 framework defines seven categories of enablers:

* Principles, Policies and Frameworks
* Processes
* Organisational Structures
* Culture, Ethics and Behaviour
* Information
* Services, Infrastructure and Applications
* People, Skills and Competencies

#### Principle 5: Separating Governance From Management

The COBIT 5 framework makes a clear distinction between governance and management. These two disciplines encompass different types of activities, require different organisational structures and serve different purposes. COBIT 5’s view on this key distinction between governance and management is:

* Governance

Governance ensures that stakeholder needs, conditions and options are evaluated to determine balanced, agreed-on enterprise objectives to be achieved; setting direction through prioritisation and decision making; and monitoring performance and compliance against agreed-on direction and objectives.

> In most enterprises, overall governance is the responsibility of the board of directors under the leadership of the chairperson. Specific governance responsibilities may be delegated to special organisational structures at an appropriate level, particularly in larger, complex enterprises.

* Management

Management plans, builds, runs and monitors activities in alignment with the direction set by the governance body to achieve the enterprise objectives.

> In most enterprises, management is the responsibility of the executive management under the leadership of the chief executive officer (CEO).




<file_sep>+++

title = "Context"
description = "Context Manager,  Introspection"
weight=8
+++


## Context managers

### context manager

* context manager an object designed to be used in a with-statement

    ```python
    with context-manager:
        body

    with context-manager:
        context-manager.begin()
        body
        context-manager.end()

    with context-manager:
        setup()
        body
        teardown()

    with context-manager:
        context-manager.begin()
        body
        context-manager.end()        

    with context-manager:
        allocation()
        body
        deallocation()

    with context-manager:
        enter()
        body
        exit()

    ```        

* A context-manager ensures that resources are properly and automatically managed
    * enter() prepares the manager for use 
    * exit() cleans it up

* Context-manager Protocol

```python
__enter__(self)

__exit__(self,
        exc_type, ## exception type 
        exc_val,  ## exception object
        exc_tb)    ## exception traceback
```

* `__enter__()`

    * called before entering with-statement body
    * return value bound to as variable
    * can return value of any type
    * commonly returns context-manager itself

* `__exit__()` called when with-statement body exits
* By default `__exit__()` propagates exceptions thrown from the with-statement’s code block
* `__exit__()` should never explicitly re-raise exceptions
* `__exit__()` should only raise exceptions if it fails itself

## contextlib

* `contextlib` : standard library module for working with context managers
* `contextmanager` is a decorator you can use to create new context managers

    ```python
    @contextlib.contextmanager
    def my_context_manager():
        ## <ENTER>
        try:
        yield [value]
        ## <NORMAL EXIT>
        except:
        ## <EXCEPTIONAL EXIT>
        raise
        with my_context_manager() as x:
        ## . . .
    ```

* `contextmanager` lets you define context-managers with simple control flow It allows you to leverage the statefulness of generators
* `contextmanager` uses standard exception handling to propagate exceptions
* `contextmanager` explicitly re-raises – or doesn’t catch – to propagate exceptions
* `contextmanager` swallows exceptions by not re-raising them
* Exceptions propagated from __inner__ context managers will be seen by __outer__ context managers
* multiple context managers with-statements can use as many context-managers as you need
* Never pass list 
    
    - Example -- The code below is the same
    
        ```python
        with cm1() as a, cm2() as b:
            BODY

        with cm1() as a:
            with cm2() as b:
                BODY
        ```

* Examples

    -- simple sample

    ```python
    import contextlib

    class LoggingContextManager:
        def __enter__(self):
            print('LoggingContextManager.__enter__()')
            return "You're in a with-block!"

        def __exit__(self, exc_type, exc_val, exc_tb):
            if exc_type is None:
                print('LoggingContextManager.__exit__: '
                    'normal exit detected')
            else:
                print('LoggingContextManager.__exit__: '
                    'Exception detected! '
                    'type={}, value={}, traceback={}'.format(
                        exc_type, exc_val, exc_tb))                     

    @contextlib.contextmanager
    def logging_context_manager():
        print('logging_context_manager: enter')
        try:
            yield "You're in a with-block!"
            print('logging_context_manager: normal exit')
        except Exception:
            print('logging_context_manager: exceptional exit',
                sys.exc_info())
            raise


    if __name__ == "__main__":
        with LoggingContextManager() as log:
            pass

        with logging_context_manager() as clog:
            pass

    ## test result
    ## LoggingContextManager.__enter__()
    ## LoggingContextManager.__exit__: normal exit detected
    ## logging_context_manager: enter
    ## logging_context_manager: normal exit


    ```

    * nested 

    ```python
            
    @contextlib.contextmanager
    def nest_test(name):
        print('Entering', name)
        yield name
        print('Exiting', name)




    if __name__ == "__main__":
        with nest_test('outer') as n1, nest_test('inner nested in ' + n1):
            pass

    ## test result
    ## Entering outer
    ## Entering inner nested in outer
    ## Exiting inner nested in outer
    ## Exiting outer
            
    ```


    * db & transaction

    ```python
    import contextlib

    class Connection:
        def __init__(self):
            self.xid = 0

        def _start_transaction(self, read_only=True):
            print('starting transaction', self.xid)
            rslt = self.xid
            self.xid = self.xid + 1
            return rslt

        def _commit_transaction(self, xid):
            print('committing transaction', xid)

        def _rollback_transaction(self, xid):
            print('rolling back transaction', xid)

    class Transaction:
        def __init__(self, conn, read_only=True):
            self.conn = conn
            self.xid = conn._start_transaction(read_only=read_only)

        def commit(self):
            self.conn._commit_transaction(self.xid)

        def rollback(self):
            self.conn._rollback_transaction(self.xid)

    @contextlib.contextmanager
    def start_transaction(connection):
        tx = Transaction(connection)

        try:
            yield tx
        except Exception:
            tx.rollback()
            raise

        tx.commit()
    ```




### Introspection

* type of object is type

* Example

    * introspector

        ```python
        import inspect
        import reprlib
        import itertools
        ## from sorted_set import SortedSet

        from bisect import bisect_left
        from collections.abc import Sequence, Set
        from itertools import chain


        class SortedSet(Sequence, Set):

            def __init__(self, items=None):
                self._items = sorted(set(items)) if items is not None else []

            def __contains__(self, item):
                try:
                    self.index(item)
                    return True
                except ValueError:
                    return False

            def __len__(self):
                return len(self._items)

            def __iter__(self):
                return iter(self._items)

            def __getitem__(self, index):
                result = self._items[index]
                return SortedSet(result) if isinstance(index, slice) else result

            def __repr__(self):
                return "SortedSet({})".format(
                    repr(self._items) if self._items else ''
                )

            def __eq__(self, rhs):
                if not isinstance(rhs, SortedSet):
                    return NotImplemented
                return self._items == rhs._items

            def __ne__(self, rhs):
                if not isinstance(rhs, SortedSet):
                    return NotImplemented
                return self._items != rhs._items

            def _is_unique_and_sorted(self):
                return all(self[i] < self[i + 1] for i in range(len(self) - 1))

            def index(self, item):
                assert self._is_unique_and_sorted()
                index = bisect_left(self._items, item)
                if (index != len(self._items)) and (self._items[index] == item):
                    return index
                raise ValueError("{} not found".format(repr(item)))

            def count(self, item):
                assert self._is_unique_and_sorted()
                return int(item in self)

            def __add__(self, rhs):
                return SortedSet(chain(self._items, rhs._items))

            def __mul__(self, rhs):
                return self if rhs > 0 else SortedSet()

            def __rmul__(self, lhs):
                return self * lhs

            def issubset(self, iterable):
                return self <= SortedSet(iterable)

            def issuperset(self, iterable):
                return self >= SortedSet(iterable)

            def intersection(self, iterable):
                return self & SortedSet(iterable)

            def union(self, iterable):
                return self | SortedSet(iterable)

            def symmetric_difference(self, iterable):
                return self ^ SortedSet(iterable)

            def difference(self, iterable):
                return self - SortedSet(iterable)

        def full_sig(method):
            try:
                return method.__name__ + str(inspect.signature(method))
            except ValueError:
                return method.__name__ + '(...)'


        def brief_doc(obj):
            doc = obj.__doc__
            if doc is not None:
                lines = doc.splitlines()
                if len(lines) > 0:
                    return lines[0]
            return ''


        def print_table(rows_of_columns, *headers):
            num_columns = len(rows_of_columns[0])
            num_headers = len(headers)
            if len(headers) != num_columns:
                raise TypeError("Expected {} header arguments, "
                                "got {}".format(num_columns, num_headers))
            rows_of_columns_with_header = itertools.chain([headers], rows_of_columns)
            columns_of_rows = list(zip(*rows_of_columns_with_header))
            column_widths = [max(map(len, column)) for column in columns_of_rows]
            column_specs = ('{{:{w}}}'.format(w=width) for width in column_widths)
            format_spec = ' '.join(column_specs)
            print(format_spec.format(*headers))
            rules = ('-' * width for width in column_widths)
            print(format_spec.format(*rules))
            for row in rows_of_columns:
                print(format_spec.format(*row))


        def dump(obj):
            print("Type")
            print("====")
            print(type(obj))
            print()

            print("Documentation")
            print("=============")
            print(inspect.getdoc(obj))
            print()

            print("Attributes")
            print("==========")
            all_attr_names = SortedSet(dir(obj))
            method_names = SortedSet(
                filter(lambda attr_name: callable(getattr(obj, attr_name)),
                    all_attr_names))
            assert method_names <= all_attr_names
            attr_names = all_attr_names - method_names
            attr_names_and_values = [(name, reprlib.repr(getattr(obj, name)))
                                    for name in attr_names]
            print_table(attr_names_and_values, "Name", "Value")
            print()

            print("Methods")
            print("=======")
            methods = (getattr(obj, method_name) for method_name in method_names)
            method_names_and_doc = [(full_sig(method), brief_doc(method))
                                    for method in methods]
            print_table(method_names_and_doc, "Name", "Description")
            print()

        if __name__ == "__main__":
            dump('a')

        ## test result
        ## Type
        ## ====
        ## <class 'str'>
        #
        ## Documentation
        ## =============
        ## str(object='') -> str
        ## str(bytes_or_buffer[, encoding[, errors]]) -> str
        ## Create a new string object from the given object. If encoding or
        ## errors is specified, then the object must expose a data buffer
        ## that will be decoded using the given encoding and error handler.
        ## Otherwise, returns the result of object.__str__() (if defined)
        ## or repr(object).
        ## encoding defaults to sys.getdefaultencoding().
        ## errors defaults to 'strict'.
        #
        ## Attributes
        ## ==========
        ## Name    Value                         
        ## ------- ------------------------------
        ## __doc__ "str(object='... to 'strict'."
        #
        ## Methods
        ## =======
        ## Name              Description                                                            
        ## ----------------- -----------------------------------------------------------------------
        ## __add__(value, /)               Return self+value.                                                     
        ## str(...)                        str(object='') -> str                                                  
        ## __contains__(key, /)            Return key in self.                                                    
        ## __delattr__(name, /)            Implement delattr(self, name).                                         
        ## __dir__(...)                    __dir__() -> list                                                      
        ## __eq__(value, /)                Return self==value.                                                    
        ## __format__(...)                 S.__format__(format_spec) -> str                                       
        ## __ge__(value, /)                Return self>=value.                                                    
        ## __getattribute__(name, /)       Return getattr(self, name).                                            
        ## __getitem__(key, /)             Return self[key].                                                      
        ## __getnewargs__(...)                                                                                    
        ## __gt__(value, /)                Return self>value.                                                     
        ## __hash__()                      Return hash(self).                                                     
        ## __init__(*args, **kwargs)       Initialize self.  See help(type(self)) for accurate signature.         
        ## __init_subclass__(...)          This method is called when a class is subclassed.                      
        ## __iter__()                      Implement iter(self).                                                  
        ## __le__(value, /)                Return self<=value.                                                    
        ## __len__()                       Return len(self).                                                      
        ## __lt__(value, /)                Return self<value.                                                     
        ## __mod__(value, /)               Return self%value.                                                     
        ## __mul__(value, /)               Return self*value.n                                                    
        ## __ne__(value, /)                Return self!=value.                                                    
        ## __new__(*args, **kwargs)        Create and return a new object.  See help(type) for accurate signature.
        ## __reduce__(...)                 helper for pickle                                                      
        ## __reduce_ex__(...)              helper for pickle                                                      
        ## __repr__()                      Return repr(self).                                                     
        ## __rmod__(value, /)              Return value%self.                                                     
        ## __rmul__(value, /)              Return self*value.                                                     
        ## __setattr__(name, value, /)     Implement setattr(self, name, value).                                  
        ## __sizeof__(...)                 S.__sizeof__() -> size of S in memory, in bytes                        
        ## __str__()                       Return str(self).                                                      
        ## __subclasshook__(...)           Abstract classes can override this to customize issubclass().          
        ## capitalize(...)                 S.capitalize() -> str                                                  
        ## casefold(...)                   S.casefold() -> str                                                    
        ## center(...)                     S.center(width[, fillchar]) -> str                                     
        ## count(...)                      S.count(sub[, start[, end]]) -> int                                    
        ## encode(...)                     S.encode(encoding='utf-8', errors='strict') -> bytes                   
        ## endswith(...)                   S.endswith(suffix[, start[, end]]) -> bool                             
        ## expandtabs(...)                 S.expandtabs(tabsize=8) -> str                                         
        ## find(...)                       S.find(sub[, start[, end]]) -> int                                     
        ## format(...)                     S.format(*args, **kwargs) -> str      
        ## format_map(...)                 S.format_map(mapping) -> str                                           
        ## index(...)                      S.index(sub[, start[, end]]) -> int                                    
        ## isalnum(...)                    S.isalnum() -> bool                                                    
        ## isalpha(...)                    S.isalpha() -> bool                                                    
        ## isdecimal(...)                  S.isdecimal() -> bool                                                  
        ## isdigit(...)                    S.isdigit() -> bool                                                    
        ## isidentifier(...)               S.isidentifier() -> bool                                               
        ## islower(...)                    S.islower() -> bool                                                    
        ## isnumeric(...)                  S.isnumeric() -> bool                                                  
        ## isprintable(...)                S.isprintable() -> bool                                                
        ## isspace(...)                    S.isspace() -> bool                                                    
        ## istitle(...)                    S.istitle() -> bool                                                    
        ## isupper(...)                    S.isupper() -> bool                                                    
        ## join(...)                       S.join(iterable) -> str                                                
        ## ljust(...)                      S.ljust(width[, fillchar]) -> str                                      
        ## lower(...)                      S.lower() -> str                                                       
        ## lstrip(...)                     S.lstrip([chars]) -> str                                               
        ## maketrans(x, y=None, z=None, /) Return a translation table usable for str.translate().                 
        ## partition(...)                  S.partition(sep) -> (head, sep, tail)                                  
        ## replace(...)                    S.replace(old, new[, count]) -> str                                    
        ## rfind(...)                      S.rfind(sub[, start[, end]]) -> int                                    
        ## rindex(...)                     S.rindex(sub[, start[, end]]) -> int                                   
        ## rjust(...)                      S.rjust(width[, fillchar]) -> str                                      
        ## rpartition(...)                 S.rpartition(sep) -> (head, sep, tail)                                 
        ## rsplit(...)                     S.rsplit(sep=None, maxsplit=-1) -> list of strings                     
        ## rstrip(...)                     S.rstrip([chars]) -> str                                               
        ## split(...)                      S.split(sep=None, maxsplit=-1) -> list of strings                      
        ## splitlines(...)                 S.splitlines([keepends]) -> list of strings                            
        ## startswith(...)                 S.startswith(prefix[, start[, end]]) -> bool                           
        ## strip(...)                      S.strip([chars]) -> str                                                
        ## swapcase(...)                   S.swapcase() -> str                                                    
        ## title(...)                      S.title() -> str                                                       
        ## translate(...)                  S.translate(table) -> str                                              
        ## upper(...)                      S.upper() -> str                                                       
        ## zfill(...)                      S.zfill(width) -> str   

        ```
<file_sep>+++


date = "2016-01-10T14:59:31+11:00"
title = "Ubuntu 14 -- desktop setup & dual boot "
description = "Post-installation for Ubuntu 14 desktop"
draft = false
+++

> *This article is mainly to help beginner install Ubuntu desktop at the first time. If you are looking for setup of Ubuntu server, please check out the blog -- [Ubuntu server setup](/blog/ubuntu-server-14/)*


## Where to install Linux?

* How to answer this quetion really depends user's computer knowledge and skills. Basically Linux can be installed on almost any PC, laptop, embedded device or tablet. So there are some suggestions for people with different level skills.*

* Beginner -- If you never install any operating system, or you never use `Unix/Linux` system, but still want to try something new. You should consider to install virtual machine on your computer and then install ubuntu on the virtual machine. [VMware](http://www.vmware.com) and [VirtualBox](https://www.virtualbox.org) are both very good products. 
    
* Intermediate -- If you have installed operating system, or you have used `Unix/Linux` system, you can install it on your old machine. or for safe side try it on virtual machine at first. 

* Expert -- You can try dual boot or multiple boot operating systems on your PC. Install 10-20 operating systems on a PC with 400GB harddisk should be alright. The only problem I encountered before some operating system can not find all proper drivers to support the all devices on your PC/laptop, such as the drivers for camera, touchpad, wifi, etc. It would take you so much time to research and try. 


### Install virtual machine

__*Let's get our hands dirty*__

* Install VirtualBox/VMWare on your computer. IMO, use VirtualBox is quite handy and save you much effort, even it is free. Because in the real environment, you will use remote tool to do your admin task instead of really handling a phyiscal machine. And you can try another Liunx OS on VirtualBox.

* If you lean to commercial product, you can choose VMWare. There is free trial option for you. 

### Which version to choose

* Ubuntu has variant versions for you to download and play. I will suggest you always pick LTS (Long Term Support) version to download. As beginner, desktop version is the best option for you to start. 

* After downlaod the [Ubuntu Desktop](https://www.ubuntu.com/download/desktop) from the internet. You will get a `ISO` file like this: ubuntu-xx.xx.x-desktop-amd64.iso, if your OS is 64 bits, or something like ubuntu-xx.xx.x-desktop-x86.iso for 32 bits.


### Install Unbuntu

* Before you install, you had better to backup anything on the device which you are going to install, and chcek your internet is working properly. 

* Create a new virtual machine within VMWare or VirtualBox. The processes on both softwares are almost the same. 
    * Assume your new virtual machine will sit in `C:\vbox` for VirtualBox or `C:\vm` for VMware 
    * Create a new machine from menu. Type in the name of vm. e.g. Unbuntu
    * Select the type of operating system: Linux
    * You can choose Ubuntu(32/64 bit ) or something else. It doesn't matter. We don't use any built-in xxx.iso files from VirtualBox or VMware. Then click `Next`
    * Select the memory size for the Unbuntu. 2G RAM is minimal requirement. I prefer up to 30% of total memory size. And then click `Next`.
    * Select "Create a virtual hard drive now", and then click `Next`.
    * Select defaut VDI, then click `Next`.
    * Select "Dynamically allocated", then click `Next`.
    * Choose the location of Unbuntu. e.g. c:\vbox\ubuntu\ubuntu.vdi or c:\vm\ubuntu\ubuntu.vmdk. Select the size of VDI\VMDK file. At least 8G. I'd like to select 16 or more. Then a Unbuntu virtual machine has been created. 
 
* Config the Unbuntu hardware setting. 

    **VirtualBox**

    * On VirtualBox toolbar, there is a `Start` button. Click `Start`, then go the Storage item. 
    * Under the Storage Tree section, there is Empty CD icon. Click the Empty icon.     
    * Under the Attributes section, click the CD icon at the end of the dropdown list of CD/DVD Drive. Choose the Unbuntu iso file which you download from Unbuntu.org. Click `OK`.
    * Leave all the other setting as default. Click the `Start`button on the toolbar. 

    **VMWare**

    * On VMware, you can find CD/DVD button on the tab page of new virtual machine. 
    * Click the CD button at the end of the dropdown list of CD/DVD Drive. Choose the Unbuntu iso file which you download from Unbuntu.org. Click `OK`.
    * Leave all the other setting as default. Click the `Power on this virtual machine` option on the tab page. 


* Ubuntu provides a friendly and beautiful UI to complete installation instead of ugly and terrified terminal, as geek's computer shown on sci-fi movie. If you choose VMWare or VirtulBox as machine, you can open the page of [installation steps](https://www.ubuntu.com/download/desktop/install-ubuntu-desktop) on your browser. You just need to follow the instruction step by step, it will take you around 1-2 hours to complete. 

## Things to do after installing Ubuntu desktop

* Ubuntu desktop is very nice and friendly, even it is different from your Windows. Basically you don't need any geek's skill to play around on Uubuntu desktop and use it as your Windows. There are tons of free software you can download from Ubuntu Software Center, so you don't worry where to find the softwre you need. Considering you are the beginner, some suggestions and caveats will be highlighted below, but none of these needs command line and terminal. 
 * Disable the system upgrade automatically to new LTS version.
 * Disable the system power manager to suspend your PC.
 * Disable the system problem report service.
 * Enbale the third party packages. 
 * Install `Utity Tweek` to help you customize your UI. 

## Things don't do

* Something below is suggested not to do, because I assume you are Ubuntu or Linux beginner. I don't wnat you to feel frustrated at the beginning of your Ubuntu desktop journey. It is the same that 99.9999% of Windows user should not delete cache files `C:\Windows` or change the system registry, expect they really understand what they are doing.     
 * Don't optimzie your memory setting. It is really not a big deal. 
 * Don't try to change your `Utity` to other Ubuntu desktop, e.g. Ubuntu MATE, Ubuntu Xface, etc.    
 * Don't follow the tips online to use `root` in terminal before you fully understand what the commands do.
 * Don't try to mount other drives on your computer, if it is mounted automatically.   


## Dual boot or multiple boot with Windows

**This section is for  people who want to install multiple operating system on the actual PC, instead of virtual machine as above. Obviously, it is not for beginner user, but everyone must experience the first time to go to higher level. For safe side, I strongly suggest you should use a old PC or redundant one to test it.**

* For dual or multiple boot, you need to make sure your disk is formatted as GPT. It will save you so much effort later to install other operating systems. 

* I suggest Windows first approach for multiple boot systems, because that is easier than the other way around. After install Windows on your PC, you need to shrink Windows disk space for other operating systems with `Disk Management`. 

* Now you need to prepare Ubuntu USB installer or DVD. Place the USB stick or DVD in the appropriate drive, reboot the machine and instruct the BIOS/UEFI to boot-up from the DVD/USB by pressing a special function key (usually F12, F10 or F2 depending on the vendor specifications).

* When you install Ubuntu, you need to select a separate `boot` drive for dual boot systems. Usually you just need to pick the drive `efi` as `boot` drive. Your drives will be supposed to format with GPT as following structure.

```ini
sda
  +----sda1        nfts               500M         Windows recovery
  +----sda2        efi       /boot    100M         grub2 , Windows boot manager
  +----sda3                  /        10M            
  +----sda4        ntfs      /      40000M         Window 7/8/10
  +----sda5                  swap   <Double size of your RAM size>   
  +----sda6        ext4      /      20000M         Ubuntu 14 desktop
  +----sda7        ext4      /      20000M         Fedora 20 desktop           
  +----sda8        ext4      /      20000M         CentOS 6 desktop
  +----sda9        ext4      /      20000M         OpenSuse desktop
  + ...
```

## Troubleshooting: Windows always boots first

**Change BIOS**
* Start your PC by pressing a pressing a special function key (usually F12, F10 or F2 depending on the vendor specifications).
* Some PC's BIOS has `BOOT` tab option, open the `BOOT` tab, you will find the `OS Boot Manager`. It is the simplest way to fix the issue. If your PC's BIOS has no such setting feature, you need to check the next section. 

**Change the Windows Boot Manager**
* Login windows with common prompt 
* Restart windows, meanwhile press shift key
* In the options page, choose change to other options
* Troubleshooting
* Command Prompt
* Login in Windows with Common prompt
* Use BCDEdit to change windows boot manager. Change to boot ubuntu at first

```bash
REM backup
bcdedit /enum > X:\Users\public\documents\bcdedit.txt
REM change the bootmgr 
bcdedit /set {bootmgr} path \EFI\ubuntu\grubx64.efi
```
* After you reboot system, you will see the Grub 2 menu as follow.

```bash
                             GNU GRUB version 2.0
---------------------------------------------------------------------------------- 
| Ubuntu 
| Advanced options for Unbuntu
| Windows Boot Manager ( on /dev/sda2 )
| Fedora 20
| Advanced options for Fedora 20
| OpenSuse 
| Advanced options for OpenSuse
| ....

```

<file_sep>+++
title = "Getting started"
description="Golang Introduction: Basic Command, Assignment operator,Array, Slice  ..."
weight=1
+++


## Go Introduction

> Golang's popularity is skyrocketing. The thriving of Docker and Kubernetes push the Golang to a higher level. 

> Go is easy to become functional with and appropriate for junior developers to work on. Also, having a language that encourages readability and comprehension is extremely useful. The mixture of duck typing (via interfaces) and convenience features such as ":=" for short variable declarations give Go the feel of a dynamically typed language while retaining the positives of a strongly typed one.

> Go's native concurrency is a boon for network applications that live and die on concurrency. Go is an explicitly engineered programming language, specifically designed with these new requirements in mind. Written expressly for the cloud, Go has been growing in popularity because of its mastery of concurrent operations and the beauty of its construction.


### Purpose 

* This Golang notes will be different from other language's note. It will include more basic Golang stuff comparing with other languages.
* Including more basic stuff, but it doesn't mean I will go through the basic types, conditions, etc. Golang website has done excellent job to explain all these clearly, including code sample as well.
* As a ployglot developer, I shift between strong type language and dynamic type language depends on projects in the last decade. My programming paradigm has been mixed, and that is why I have notes to remind myself the best practices for different languages.   
* As I mentioned above, I will only highlight the feature and best practice of Golang different from other languages I am familiar with.


### Basic Command

* Install Golang
* Setup the environment variables properly

    GOOS=linux
    GOROOT=/usr/local/go
    GOARCH=amd64
    GOPATH=/home/<user_account>/ws/go
    GOBIN=/home/<user_account>/ws/go/bin


* You can find the Golang helloworld program from the home page
* Save the program to the location `$GOPATH/src`
* Build, Run & Install the helloworld program `hello.go`

```bash
# Navigate to hello.go location
cd $GOPATH/src

# Build the hello.go
go build hello.go
#  Run the executable program
./hello.go

# Build and Run in one command
go run hello.go

# Install the program 
go install hello.go
hello

```
### Assignment Operator

* The `:=` operator effectively makes a new variable; it is also called an initializing declaration.This is the preferred form, but it can only be used inside functions, not in package scope. 

```go
// Multiple declarations on a single line
a, b, c := 5, 7, “abc”
```
### Parallel or Simultaneous assignment 

* There is no need to make a swap function in Go

```go
// Following code perform a swap function
a,b = b,a
```

### Powerful blank variable

* The blank identifier _ can also be used to throw away values. _ is in effect a write-only variable, you cannot ask for its value. It exists because a declared variable in Go must also be used, and sometimes you don’t need to use all return values from a function.

```go
_, b = "abc", 7
// A function return val and error, but error can be ignored.

val, _ = FuncReturnValAndErr(' Ignore the return error ')
```

### Array & Slice

#### Array

> An array is a numbered and fixed-length sequence of data items (elements) of the same single type; The length must be a constant expression, that must evaluate to a non-negative integer value.

* Create an array variable

```go
var arr1 [5]int;

for  i := range arr1 { 
    fmt.Printf(" index = %d , val = %d \n", i, arr1[i])
}

// ----------- Output ---------------
//  index = 0 , value = 0 
//  index = 1 , value = 0 
//  index = 2 , value = 0 
//  index = 3 , value = 0 
//  index = 4 , value = 0 
```

* Create an arrary with Array literal

```go
// Print out the array below will get the same output as arr1 above
var arr2 = [...]int{0,0,0,0,0};
var arr3 = []int{0,0,0,0,0}; // This is a slice instead of array

```

#### Slice

* A slice is a reference to a contiguous segment(section) of an array (which we will call the underlying array, and which is usually anonymous), so a slice is a reference type (thus more akin to the array type in C/C++, or the list type in Python).


```go

arr := [5]int {1,2,3,4,5}
slice := arr[0:2]
fmt.Printf( " slice = %v \n", slice)
slice = arr[0:]
fmt.Printf( " slice = %v \n", slice)
slice = arr[:2]
fmt.Printf( " slice = %v \n", slice)
slice = arr[1:5]
fmt.Printf( " slice = %v \n", slice)
slice = arr[:5]
fmt.Printf( " slice = %v \n", slice)

// ---- output -----
//  slice = [1 2] 
//  slice = [1 2 3 4 5] 
//  slice = [1 2] 
//  slice = [2 3 4 5] 
//  slice = [1 2 3 4 5] 

```


#### Re-slicing

* Changing the length of the slice is called re-slicing, it is done e.g. like: `slice1 = slice1[0:end]` where end is another end-index (length) than before. 

* Resizing a slice by 1 can be done as follows: `sl = sl[0:len(sl)+1] // extend length by 1`

* A slice can be resized until it occupies the whole underlying array.

#### Copy and append slices

```go
package main
import “fmt”
func main() {

    sl_from := []int{1,2,3}
    sl_to := make([]int,10)
    n := copy(sl_to, sl_from)
    fmt.Println(sl_to)
    // output: [1 2 3 0 0 0 0 0 0 0]

    fmt.Printf("Copied %d elements\n", n) // n == 3

    sl3 := []int{1,2,3}
    sl3 = append(sl3, 4, 5, 6)
    fmt.Println(sl3)
}
```


### make() and new()

* new(T) allocates zeroed storage for a new item of type T and returns its address, a value of type *T: it returns a pointer to a newly allocated zero value of type T, ready for use; it applies to value types like arrays and structs; it is equivalent to &T{ }

* make(T) returns an initialized value of type T; it applies only to the 3 built-in reference types: slices, maps and channels.

* In other words, new allocates; make initializes.

```go
var p *[]int = new([]int) // *p == nil; with len and cap 0
// or use assignment operator
p := new([]int)

var v []int = make([]int, 10, 50)
// or use assignment operator
v := make([]int, 10, 50)

```



### Good practices

#### Use Buffer to concat string

* Go has a package bytes with manipulation functions for that kind of type Buffer. The sample below is much more memory and CPU-efficient than +=, especially if the number of strings to concatenate is large.

```go
var buffer bytes.Buffer
    for {
        if s, ok := getNextString(); ok { //method getNextString() not shown here
            buffer.WriteString(s)
        } else {
            break
    }
}

fmt.Print(buffer.String(), “\n”)

```

#### Substr 

Slice makes substringing more easier than other language.  `subStr := str[start:end]`


#### Change a character in string 

* To do this you first have to convert the string to an array of bytes, then an array-item of a certain index can be changed, and then the array must be converted back to a new string.

```go
str := "golang"
chars:= []byte(str)
chars[1] = '0'
newStr := string(chars) // g0lang
```






<file_sep>+++
title = "DigitialOcean: Droplet"
description="Droplet Introduction"
weight=1
+++



## Droplet

DigitalOcean Droplets are Linux-based virtual machines (VMs) that run on top of virtualized hardware. Each Droplet you create is a new server you can use, either standalone or as part of a larger, cloud-based infrastructure.


### Prerequisite

* Prepare a bank account or credit card
* Signup DigitalOcean account and activate it
* Create a new prject

### OS Options

* Ubuntu
* FreeBSD
* Fedora
* Debian
* CentOS

### Plans and Pricing
We offer four different kinds of Droplet plans: one shared CPU plan and three dedicated CPU plans.

Droplet Plan|	CPU | 	Range of Resources |	RAM-to-CPU Ratio|	Processor
----|----|----|----|----
Standard |	Shared |	1 - 32 vCPUs 1 - 192 GB | RAM	Variable	| 
General | Purpose	Dedicated| 	2 - 40 vCPUs 8 - 160 GB RAM	| 4 GB per vCPU	| Intel Xeon Skylake (2.7 GHz, 3.7 GHz turbo)
CPU-Optimized	| Dedicated	| 2 - 32 vCPUs| 4 - 64 GB RAM	2 GB per vCPU	| Intel Xeon Broadwell (2.6 GHz) Intel Xeon Skylake (2.7 GHz, 3.7 GHz turbo)
Memory-Optimized	|Dedicated|	2 - 32 vCPUs| 16 - 256 GB RAM	8 GB per vCPU	|


### Create a droplet

* Create a droplet
* Choose the operating system you prefer
* Pick up the CPU and RAM you need.
  * The lowest one is 1 vCPU and 1G RAM
  * The pricing for above setting is $5/month
* Choose the region your droplet will install
* Always use SSH
  * Create a new SSH key via `ssh-keygen`
  * Add the publc key to your droplet 
* Click the button "Create Droplet"


### Access Droplet

#### Get Public IP

> After you create your droplet, you will see a list of droplests as follow

Name | IP Address | Created | Tags
----|----|---|---
droplet-name | xxx.xxx.xxx.xxx | 10 mins ago | 


#### Login via SSH

```
ssh -i ~/.ssh/<your_droplet_rsa> root@<xxx.xxx.xxx.xxx>
```

#### Congrats

* Your droplet is up and running
* Next we will continue the setup droplet as web server.<file_sep>+++
title = "Project, Vector, String & Hashmap"
description="Rustlang Introduction: Project management, Vector, String and Hashmap"
weight = 2
+++

### Project management

* Rust has a number of features that allow you to manage your code’s organization, including which details are exposed, which details are private, and what names are in each scope in your programs.

    * Packages: A Cargo feature that lets you build, test, and share crates
    * Crates: A tree of modules that produces a library or executable
    * Modules and use: Let you control the organization, scope, and privacy of paths
    * Paths: A way of naming an item, such as a struct, function, or module

#### Package & Crate

*  A package is one or more crates that provide a set of functionality. A package contains a Cargo.toml file that describes how to build those crates.

* A crate will group related functionality together in a scope so the functionality is easy to share between multiple projects.


#### Module

* Modules let us organize code within a crate into groups for readability and easy reuse. Modules also control the privacy of items, which is whether an item can be used by outside code (public) or is an internal implementation detail and not available for outside use (private).

#### Path

* A path can take two forms:

    * An absolute path starts from a crate root by using a crate name or a literal crate.
    * A relative path starts from the current module and uses self, super, or an identifier in the current module.


### Vector

* Vectors allow you to store more than one value in a single data structure that puts all the values next to each other in memory. 

    ```rs
    let v = vec![1, 2, 3, 4, 5];

    let third: &i32 = &v[2];
    println!("The third element is {}", third);

    match v.get(2) {
        Some(third) => println!("The third element is {}", third),
        None => println!("There is no third element."),
    }
    ```

#### Iterating over the Values

* Immutable

    ```rs
    let v = vec![100, 32, 57];
    for i in &v {
        println!("{}", i);
    }
    ```

* Mutable

    ```rs
    let mut v = vec![100, 32, 57];
    for i in &mut v {
        *i += 50;
    }
    ```

### String 

* Rust has only one string type in the core language, which is the string slice str that is usually seen in its borrowed form &str. 

* The String type, which is provided by Rust’s standard library rather than coded into the core language, is a growable, mutable, owned, UTF-8 encoded string type. When Rustaceans refer to “strings” in Rust, they usually mean the String and the string slice &str types, not just one of those types. 


* Rust’s standard library also includes a number of other string types, such as OsString, OsStr, CString, and CStr. Library crates can provide even more options for storing string data.

#### New string

* use new function

```rs
let mut s = String::new();
```

#### Updating a String

* A String can grow in size and its contents by using the push_str method to append a string slice.

* The push method takes a single character as a parameter and adds it to the String. 

```rs
let mut s1 = String::from("foo");
let s2 = "bar";
s1.push_str(s2);
println!("s1 is {}", s1);
println!("s2 is {}", s2);
   
let mut s3 = String::from("lo");
s3.push('l'); 
println!("s3 is {}", s3);
```

* Concatenation with the + Operator or the format! Macro 

```rs
let s1 = String::from("Hello, ");
let s2 = String::from("world!");
let s3 = s1 + &s2; // note s1 has been moved here and can no longer be used
```
* Iterates char of string

```rs
for c in "नमस्ते".chars() {
    println!("{}", c);
}

for b in "नमस्ते".bytes() {
    println!("{}", b);
}
```

#### Strings Are Not So Simple

* To summarize, strings are complicated. Different programming languages make different choices about how to present this complexity to the programmer. Rust has chosen to make the correct handling of String data the default behavior for all Rust programs, which means programmers have to put more thought into handling UTF-8 data upfront. 


### Hashmap

* The type HashMap<K, V> stores a mapping of keys of type K to values of type V. It does this via a hashing function, which determines how it places these keys and values into memory. 

* Hash maps are useful when you want to look up data not by using an index, as you can with vectors, but by using a key that can be of any type. 

#### New hashmap

* create an empty hash map with new and add elements with insert.

```rs
use std::collections::HashMap;

let mut scores = HashMap::new();

scores.insert(String::from("Blue"), 10);
scores.insert(String::from("Yellow"), 50);
```

* Another way of constructing a hash map is by using the collect method on a vector of tuples, where each tuple consists of a key and its value. The collect method gathers data into a number of collection types, including HashMap.

```rs
use std::collections::HashMap;

let teams  = vec![String::from("Blue"), String::from("Yellow")];
let initial_scores = vec![10, 50];

let scores: HashMap<_, _> = teams.iter().zip(initial_scores.iter()).collect();
```

#### Hash Maps and Ownership

* For types that implement the Copy trait, like i32, the values are copied into the hash map. For owned values like String, the values will be moved and the hash map will be the owner of those values.

```rs
use std::collections::HashMap;

let field_name = String::from("Favorite color");
let field_value = String::from("Blue");

let mut map = HashMap::new();
map.insert(field_name, field_value);

println!("field_name: {} field_value: {}", field_name, field_value);
// error[E0382]: borrow of moved value: `field_name`
// error[E0382]: borrow of moved value: `field_value`
```

#### Accessing Values in a Hash Map

*  get a value out of the hash map by providing its key to the get method

```rs
use std::collections::HashMap;

let mut scores = HashMap::new();

scores.insert(String::from("Blue"), 10);
scores.insert(String::from("Yellow"), 50);

for (key, value) in &scores {
    println!("{}: {}", key, value);
}
```

#### Updating a Hash Map

__Overwrites value__

If we insert a key and a value into a hash map and then insert that same key with a different value, the value associated with that key will be replaced. 

```rs

use std::collections::HashMap;

let mut scores = HashMap::new();

scores.insert(String::from("Blue"), 10);
scores.insert(String::from("Blue"), 25);

println!("{:?}", scores);
```

__Only Inserting a Value If the Key Has No Value__

* It’s common to check whether a particular key has a value and, if it doesn’t, insert a value for it. Hash maps have a special API for this called entry that takes the key you want to check as a parameter. The return value of the entry method is an enum called Entry that represents a value that might or might not exist.

```rs
#![allow(unused_variables)]
fn main() {
    use std::collections::HashMap;

    let mut scores = HashMap::new();
    scores.insert(String::from("Blue"), 10);

    scores.entry(String::from("Yellow")).or_insert(50);
    scores.entry(String::from("Blue")).or_insert(50);

    println!("{:?}", scores);
    // {"Blue": 10, "Yellow": 50}
}
```

__Updating a Value Based on the Old Value__

Another common use case for hash maps is to look up a key’s value and then update it based on the old value. 

```rs
use std::collections::HashMap;

let text = "hello world wonderful world";

let mut map = HashMap::new();

for word in text.split_whitespace() {
    let count = map.entry(word).or_insert(0);
    *count += 1;
}

println!("{:?}", map);
```


### Hashing Functions

* By default, HashMap uses a “cryptographically strong”1 hashing function that can provide resistance to Denial of Service (DoS) attacks. This is not the fastest hashing algorithm available, but the trade-off for better security that comes with the drop in performance is worth it. 

* A hasher is a type that implements the BuildHasher trait.<file_sep>+++
title = "F# Computations 2"
description = "F# Computation - Bind, Return, Map & Apply"
weight = 11
+++


## Map

Common Names: map, fmap, lift, Select

Common Operators: `<$> <!>`

What it does: Lifts a function into the elevated world

Signature: `(a->b) -> E<a> -> E<b>`. 

Alternatively with the parameters reversed: `E<a> -> (a->b) -> E<b>`

Description
“map” is the generic name for something that takes a function in the normal world and transforms it into a corresponding function in the elevated world.

An alternative interpretation of map is that it is a two parameter function that takes an elevated value `E<a>` and a normal function `a->b`, and returns a new elevated value `E<b>` generated by applying the function a->b to the internal elements of `E<a>`.

```fsharp
/// map for Options
let mapOption f opt =
    match opt with
    | None ->
        None
    | Some x ->
        Some (f x)
// has type : ('a -> 'b) -> 'a option -> 'b option

/// map for Lists
let rec mapList f list =
    match list with
    | [] ->
        []
    | head::tail ->
        // new head + new tail
        (f head) :: (mapList f tail)
// has type : ('a -> 'b) -> 'a list -> 'b list
```

## Return

Common Names: return, pure, unit, yield, point

Common Operators: None

What it does: Lifts a single value into the elevated world

Signature: `a -> E<a>`

Description
“return” (also known as “unit” or “pure”) simply creates an elevated value from a normal value.

This function goes by many names, but I’m going to be consistent and call it return as that is the common term for it in F#, and is the term used in computation expressions.

```fsharp 
// A value lifted to the world of Options
let returnOption x = Some x
// has type : 'a -> 'a option

// A value lifted to the world of Lists
let returnList x  = [x]
// has type : 'a -> 'a list
```


## Apply

Common Names: apply, ap

Common Operators: <*>

What it does: Unpacks a function wrapped inside an elevated value into a lifted function `E<a> -> E<b>`

Signature: `E<(a->b)> -> E<a> -> E<b>`

Description
“apply” unpacks a function wrapped inside an elevated value (E<(a->b)>) into a lifted function `E<a> -> E<b>`

An alternative interpretation of apply is that it is a two parameter function that takes an elevated value `(E<a>)` and an elevated function `(E<(a->b)>)`, and returns a new elevated value `(E<b>)` generated by applying the function `a->b` to the internal elements of `E<a>`.

For example, if you have a one-parameter function `(E<(a->b)>)`, you can apply it to a single elevated parameter to get the output as another elevated value.

If you have a two-parameter function `(E<(a->b->c)>)`, you can use apply twice in succession with two elevated parameters to get the elevated output.



```fsharp
module Option =

    // The apply function for Options
    let apply fOpt xOpt =
        match fOpt,xOpt with
        | Some f, Some x -> Some (f x)
        | _ -> None

module List =

    // The apply function for lists
    // [f;g] apply [x;y] becomes [f x; f y; g x; g y]
    let apply (fList: ('a->'b) list) (xList: 'a list)  =
        [ for f in fList do
          for x in xList do
              yield f x ]

let add x y = x + y 


let resultOption =
    let (<*>) = Option.apply
    (Some add) <*> (Some 2) <*> (Some 3)
// resultOption = Some 5

let resultList =
    let (<*>) = List.apply
    [add] <*> [1;2] <*> [10;20]
// resultList = [11; 21; 12; 22]

```

## Apply vs. Map
The combination of apply and return is considered “more powerful” than map, because if you have apply and return, you can construct map from them, but not vice versa.

Here’s how it works: to construct a lifted function from a normal function, just use return on the normal function and then apply. This gives you the same result as if you had simply done map in the first place.

```fsharp
let resultOption2 =
    let (<!>) = Option.map
    let (<*>) = Option.apply
    add <!> (Some 2) <*> (Some 3)
// resultOption2 = Some 5

let resultList2 =
    let (<!>) = List.map
    let (<*>) = List.apply
    add <!> [1;2] <*> [10;20]
// resultList2 = [11; 21; 12; 22]

let batman =
    let (<!>) = List.map
    let (<*>) = List.apply
    // string concatenation using +
    (+) <!> ["bam"; "kapow"; "zap"] <*> ["!"; "!!"]

// result =
// ["bam!"; "bam!!"; "kapow!"; "kapow!!"; "zap!"; "zap!!"]
```




## Zip, ZipList
Common Names: zip, zipWith, map2

Common Operators: `<*>` (in the context of ZipList world)

What it does: Combines two lists (or other enumerables) using a specified function

Signature: `E<(a->b->c)> -> E<a> -> E<b> -> E<c>` where E is a list or other enumerable type, or `E<a> -> E<b> -> E<a,b>` for the tuple-combined version.

Description
Some data types might have more than one valid implementation of apply. For example, there is another possible implementation of apply for lists, commonly called ZipList or some variant of that.

In this implementation, the corresponding elements in each list are processed at the same time, and then both lists are shifted to get the next element. That is, the list of functions [f; g] applied to the list of values [x; y] becomes the two-element list [f x; g y]


```fsharp 

// alternate "zip" implementation
// [f;g] apply [x;y] becomes [f x; g y]
let rec zipList fList xList  =
    match fList,xList with
    | [],_
    | _,[] ->
        // either side empty, then done
        []
    | (f::fTail),(x::xTail) ->
        // new head + new tail
        (f x) :: (zipList fTail xTail)
// has type : ('a -> 'b) -> 'a list -> 'b list

```

## Bind

Called for let! and do! in computation expressions.

```
M<'T> * ('T -> M<'U>) -> M<'U>
```

The `let!` keyword binds the result of a call to another computation expression to a name. let! is defined by the Bind(x, f) member on the builder type.

The `do!` keyword is for calling a computation expression that returns a unit-like type (defined by the Zero member on the builder)

- Bind is also actually functional concepts than functions
- Bind is about lifting a function input that to an "Elavated" world
- Bind is similar to Linq's SelectMany method

Common Names: bind, flatMap, andThen, collect, SelectMany

Common Operators: `>>=` (left to right), `=<<` (right to left )

What it does: Allows you to compose world-crossing (“monadic”) functions

Signature: `(a->E<b>) -> E<a> -> E<b>`

Alternatively with the parameters reversed: `E<a> -> (a->E<b>) -> E<b>`

Description
We frequently have to deal with functions that cross between the normal world and the elevated world.

An alternative interpretation of bind is that it is a two parameter function that takes a elevated value `E<a>` and a “monadic function” `a -> E<b>`, and returns a new elevated value `E<b>` generated by “unwrapping” the value inside the input, and running the function `a -> E<b>` against it.

For example: a function that parses a string to an int might return an `Option<int>` rather than a normal int, a function that reads lines from a file might return `IEnumerable<string>`, a function that fetches a web page might return Async<string>, and so on.

These kinds of “world-crossing” functions are recognizable by their signature `a -> E<b>`; their input is in the normal world but their output is in the elevated world. Unfortunately, this means that these kinds of functions cannot be linked together using standard composition.


```fsharp
type Int32 with 
    static member ParseAsOption str = 
        match Int32.TryParse (str:string) with
        | false, _ -> None
        | true, x -> Some x

let bindOption f opt = 
    match opt with 
    | Some x -> f x
    | None -> None

let joinOption opt = 
    match opt with
    | Some innerOpt -> innerOpt
    | None -> None

let bindOption2 f opt = joinOption (Option.map f opt)


let input1 = Some "abcd" |> bindOption Int32.ParseAsOption
printfn "%A" input1
// None

let input2 = Some "100" |> bindOption2 Int32.ParseAsOption
printfn "%A" input3
// Some 100

let input3 = Some "200" |> Option.map Int32.ParseAsOption
printfn "%A" input3
// Some (Some 200)

let (>>=) m f = Option.bind f m

let add x y = x + y 

let liftedAdd = Some add

let apply f m = 
    match f, m with
    | Some f, Some m -> Some(f m)
    | _ -> None


let (<*>) = apply


let input4 = 
    liftedAdd <*>
        (Some "100" >>= Int32.ParseAsOption) <*>
        (Some "200" >>= Int32.ParseAsOption) 

printfn "%A" input4


```












<file_sep>+++
title="AWS: Labs"
description="Labs - "
weight=90
draft=true
+++

## Labs 1 

AWS has tons of services available on its resources panel. By reading the document or watching the online tutorial is a simple way to learn key concepts, but only the hands-on approach can guarantee you really master the knowledge and become a known-how AWS expert.

### Prerequisite

* Have an AWS account
* Install AWS Cli tool
* Under how to use AWS Cli
* Most of labs below are executed with AWS Cli (Version 1)

### Upgrade the AWS Cli to latest version

```bash
# Mac OS
pip3 install awscli --upgrade --user

```

### TODO - Lab 1

* Create a VPC  - 10.100.0.0/16 
* Create a private subnet and a public subnet
* Create a wp site with mysql db
* Test the wp


```
VPC=<vpc_name>

aws 


## Attach IGW
aws ec2 attach-internet-gateway \
--vpc-id "vpc-0d6d50e43fbfb23ff" \
--internet-gateway-id "igw-00da6dd9fc8648a25" \
--region ap-southeast-2

```


### TODO - Lab 2

* Move EC2 from VPC 1 to VPC 2

```bash
aws ec2 create-image --instance-id i-04f9dcfadd1767427 --name "NSW_PROD_REDIRECT_Last" --description "NSW_PROD_REDIRECT_Last_Large"


```
<file_sep>+++
title = "Rustlang"
description = "Rustlang Notes"
+++

{{%children style="card" description="true" sort="Weight" %}}<file_sep>+++
title = "AWS: S3 - 3"
description = "How to allow specific  VPC endpoints or IP addresses to access S3 bucket"
weight=7
+++

## Use Case

### Problem

Block all traffic to my Amazon Simple Storage Service (Amazon S3) bucket unless the traffic is from specific Amazon Virtual Private Cloud (VPC) endpoints or certain external IP addresses.



### Resolution

Use a bucket policy to specify which VPC endpoints or external IP addresses can access the S3 bucket.

> Note: An external IP address is a public IP address that can be from within a VPC or outside of a VPC. For example, an external IP address can be an Amazon Elastic Compute Cloud (Amazon EC2) instance's Elastic IP address, or the IP address of a VPC's NAT gateway or proxy server.

For example, the following bucket policy blocks traffic to the bucket unless the request is from specified VPC endpoints (aws:sourceVpce) or external IP addresses (aws:SourceIp). Note the following:

* To use this policy with the aws:sourceVpce condition, you must have a VPC endpoint for Amazon S3 attached to the route table of the EC2 instance's subnet. The VPC endpoint must be in the same AWS Region as the bucket.

* To allow users to perform S3 actions on the bucket from the VPC endpoints or IP addresses, you must explicitly allow the user-level permissions. You can explicitly allow user-level permissions on either an AWS Identity and Access Management (IAM) policy or another statement in the bucket policy.


__Warning__: This example bucket policy explicitly denies access to any requests outside the allowed VPC endpoints or IP addresses.


```json
{
  "Version": "2012-10-17",
  "Id": "VPCe and SourceIP",
  "Statement": [{
    "Sid": "VPCe and SourceIP",
    "Effect": "Deny",
    "Principal": "*",
    "Action": "s3:*",
    "Resource": [
      "arn:aws:s3:::awsexamplebucket",
      "arn:aws:s3:::awsexamplebucket/*"
    ],
    "Condition": {
      "StringNotEquals": {
        "aws:sourceVpce": [
          "vpce-1111111",
          "vpce-2222222"
        ]
      },
      "NotIpAddress": {
        "aws:SourceIp": [
          "11.11.11.11/32",
          "22.22.22.22/32"
        ]
      }
    }
  }]
}
```


Allow specific users (within the same AWS account) access to the bucket even if the users aren't sending requests from the allowed VPC endpoints or IP addresses



```json
"StringNotLike": {
    "aws:userId": [
    "AROAEXAMPLEID:*",
    "AIDAEXAMPLEID",
    "111111111111"
    ]
}
```

* AROAEXAMPLEID is the role ID of an IAM role that you want to allow
  
* AIDAEXAMPLEID is the user ID of an IAM user that you want to allow

* 111111111111 is the AWS account ID of the bucket, which represents the account's root credentials<file_sep>+++

title = "C# Console App"
description = "How to create C# console application without Visual Studio"
weight=10

+++

> __C# is an elegant and type-safe object-oriented language that enables developers to build a variety of secure and robust applications that run on the .NET Framework. C# syntax is highly expressive, yet it is also simple and easy to learn. The curly-brace syntax of C# will be instantly recognizable to anyone familiar with C, C++ or Java.__


Here I am going to demonstrate how to create simple .net project without Visual Studio 

### Prerequisites

* .Net Framework has been install to your PC or laptop, and the version of .Net Framework on your PC is 2.0+ or later
* Assume the path  of .Net Frameowork is __c:\Windows\Microsfot.Net\Frameowork\v2.0.50727__



### Create a project

* Create a project named `csharp-project` 

    ```bat
    md csharp-project
    cd csharp-project 
    md bin src
    echo.>csharp-project.proj
    echo.>src\helloworld.cs
    ```

### Update config file 

* Update config file `csharp-project.proj` as below

```xml
<Project DefaultTargets = "Compile"
    xmlns="http://schemas.microsoft.com/developer/msbuild/2003" >

    <!-- Set the application name as a property -->
    <PropertyGroup>
        <appname>csharp-app</appname>
    </PropertyGroup>

    <!-- Specify the inputs by type and file name -->
    <ItemGroup>
        <CSFile Include = "src\helloworld.cs"/>
    </ItemGroup>

    <Target Name = "Compile">
        <!-- Run the Visual C# compilation using input files of type CSFile -->
        <CSC
            Sources = "@(CSFile)"
            OutputAssembly = "bin\$(appname).exe">
            <!-- Set the OutputAssembly attribute of the CSC task
            to the name of the executable file that is created -->
            <Output
                TaskParameter = "OutputAssembly"
                ItemName = "EXEFile" />
        </CSC>
        <!-- Log the file name of the output file -->
        <Message Text="The output file is @(EXEFile)"/>
    </Target>
</Project>

```


### Create the main program

* Create a new file named `HelloWorld.cs` and copy following coding to the new file

    ```csharp
    public class Hello
    {
        public static void Main()
        {
            System.Console.WriteLine("Hello, World!");
        }
    }
    ```

### Run the console app

* Compile with MSBuild & Run 

    ```bat
    c:\Windows\Microsfot.Net\Frameowork\v2.0.50727\MSBuild
    bin\csharp-app.exe
    ```






<file_sep>+++
title = "AWS: EKS - 4"
description = "VPC, Networking"
weight=14
+++


## EKS - Part 4

### VPC Tagging

* Key: The <cluster-name> value matches your Amazon EKS cluster's name.
* Value: The shared value allows more than one cluster to use this VPC.

Key	| Value
---|----
kubernetes.io/cluster/\<cluster-name\>| shared



### Load Balancing

Amazon EKS supports the Network Load Balancer and the Classic Load Balancer for pods running on Amazon EC2 instance worker nodes through the Kubernetes service of type LoadBalancer. Classic Load Balancers and Network Load Balancers are not supported for pods running on AWS Fargate (Fargate).

* All subnets (public and private) should have this tag.

Key	| Value
---|----
kubernetes.io/cluster/\<cluster-name\>| shared




* Public subnet tagging

Key	| Value
---|----
kubernetes.io/role/elb| 1

* Private subnet tagging


Key	| Value
---|----
kubernetes.io/role/internal-elb| 1



### ALB Ingress Controller


The AWS ALB Ingress Controller for Kubernetes is a controller that triggers the creation of an Application Load Balancer (ALB) and the necessary supporting AWS resources whenever an Ingress resource is created on the cluster with the kubernetes.io/ingress.class: alb annotation. The Ingress resource configures the ALB to route HTTP or HTTPS traffic to different pods within the cluster. The ALB Ingress Controller is supported for production workloads running on Amazon EKS clusters.


* To ensure that your Ingress objects use the ALB Ingress Controller, add the following annotation to your Ingress specification. 

        annotations:
            kubernetes.io/ingress.class: alb


* Create an IAM OIDC provider and associate it with your cluster. 

        
        CLUSTER_NAME="pg-prd"
        REGION_CODE="ap-southeast-2"
        eksctl utils associate-iam-oidc-provider \
            --region  ${REGION_CODE}   \
            --cluster  ${CLUSTER_NAME}  \
            --approve

* Create an IAM policy called __ALBIngressControllerIAMPolicy__ for the ALB Ingress Controller pod that allows it to make calls to AWS APIs on your behalf. 

        aws iam create-policy \
            --policy-name ALBIngressControllerIAMPolicy \
            --policy-document https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.4/docs/examples/iam-policy.json


* Create a Kubernetes service account named __alb-ingress-controller__ in the kube-system namespace, a cluster role, and a cluster role binding for the ALB Ingress Controller to use with the following command.

        kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.4/docs/examples/rbac-role.yaml

* Create an IAM role for the ALB ingress controller and attach the role to the service account created in the previous step. 


        CLUSTER_NAME="pg-prd"
        REGION_CODE="ap-southeast-2"
        eksctl create iamserviceaccount \
            --region ${REGION_CODE} \
            --name alb-ingress-controller \
            --namespace kube-system \
            --cluster ${CLUSTER_NAME}  \
            --attach-policy-arn arn:aws:iam::202756970286:policy/ALBIngressControllerIAMPolicy \
            --override-existing-serviceaccounts \
            --approve



* Deploy the ALB Ingress Controller 
  
        kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.4/docs/examples/alb-ingress-controller.yaml

        kubectl get clusterroles | grep aws-alb-ingress-controller

* Add a line for the cluster name after the __--ingress-class=alb__ line.
  
        spec:
        containers:
        - args:
            - --ingress-class=alb
            - --cluster-name=prod
            - --aws-vpc-id=vpc-03468a8157edca5bd
            - --aws-region=region-code


* Log the ingress controller

        kubectl logs -n kube-system   deployment.apps/alb-ingress-controller

* Deploy a sample application

        kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.4/docs/examples/2048/2048-namespace.yaml
        kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.4/docs/examples/2048/2048-deployment.yaml
        kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.4/docs/examples/2048/2048-service.yaml
        kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/aws-alb-ingress-controller/v1.1.4/docs/examples/2048/2048-ingress.yaml


* Play the game on browser

        http://07f34453-2048game-2048ingr-6fa0-1986376393.ap-southeast-2.elb.amazonaws.com/
<file_sep>+++
title = "NeoVim"
description = "Empower the vi with NeoVimn"
+++

__This article will show you how to empower your vi with NeoVim.__

## NeoVim

> [Neovim](https://github.com/neovim/neovim) is a refactor, and sometimes redactor, in the tradition of Vim (which itself derives from Stevie). It is not a rewrite but a continuation and extension of Vim. Many clones and derivatives exist, some very clever—but none are Vim. Neovim is built for users who want the good parts of Vim, and more.


### Install Neovim

Please follow the installation instructions [here](https://github.com/neovim/neovim/wiki/Installing-Neovim) 

### Vim Plugin

Please follow the installation instructions from [Vim-Plug's README](https://github.com/neovim/neovim)https://github.com/junegunn/vim-plug


### Init


- Create a file `init.vim`

```lua
:set number
:set relativenumber
:set autoindent
:set tabstop=4
:set shiftwidth=4
:set smarttab
:set softtabstop=4
:set mouse=a

call plug#begin()

Plug 'http://github.com/tpope/vim-surround' " Surrounding ysw)
Plug 'https://github.com/preservim/nerdtree' " NerdTree
Plug 'https://github.com/tpope/vim-commentary' " For Commenting gcc & gc
Plug 'https://github.com/vim-airline/vim-airline' " Status bar
Plug 'https://github.com/lifepillar/pgsql.vim' " PSQL Pluging needs :SQLSetType pgsql.vim
Plug 'https://github.com/ap/vim-css-color' " CSS Color Preview
Plug 'https://github.com/rafi/awesome-vim-colorschemes' " Retro Scheme
Plug 'https://github.com/neoclide/coc.nvim'  " Auto Completion
Plug 'https://github.com/ryanoasis/vim-devicons' " Developer Icons
Plug 'https://github.com/tc50cal/vim-terminal' " Vim Terminal
Plug 'https://github.com/preservim/tagbar' " Tagbar for code navigation
Plug 'https://github.com/terryma/vim-multiple-cursors' " CTRL + N for multiple cursors
Plug 'https://github.com/EdenEast/nightfox.nvim' " A highly customizable theme
Plug 'ctrlpvim/ctrlp.vim'
Plug 'kassio/neoterm'



set encoding=UTF-8

call plug#end()

nnoremap <C-f> :NERDTreeFocus<CR>
nnoremap <C-n> :NERDTree<CR>
nnoremap <C-t> :NERDTreeToggle<CR>
nnoremap <C-l> :call CocActionAsync('jumpDefinition')<CR>

nmap <F8> :TagbarToggle<CR>

:set completeopt-=preview " For No Previews

:colorscheme nightfox

let g:NERDTreeDirArrowExpandable="+"
let g:NERDTreeDirArrowCollapsible="~"

" --- Just Some Notes ---
" :PlugClean :PlugInstall :UpdateRemotePlugins
"
" :CocInstall coc-python
" :CocInstall coc-clangd
" :CocInstall coc-snippets
" :CocCommand snippets.edit... FOR EACH FILE TYPE

" air-line
let g:airline_powerline_fonts = 1

if !exists('g:airline_symbols')
    let g:airline_symbols = {}
endif

" airline symbols
let g:airline_left_sep = ''
let g:airline_left_alt_sep = ''
let g:airline_right_sep = ''
let g:airline_right_alt_sep = ''
let g:airline_symbols.branch = ''
let g:airline_symbols.readonly = ''
let g:airline_symbols.linenr = ''

inoremap <expr> <Tab> pumvisible() ? coc#_select_confirm() : "<Tab>"

```


- Install plugins

  - Open nvim
  - Enter command mode and enter `PlugInstall`





### Trouble shooting

#### Fonts

- Mac
  - Install Nerd fonts 
    - [Follow the instruction to install Powerline Nerd Font](https://webinstall.dev/nerdfont/)  
  - Download other fonts, e.g. Caskaydia Cove Nerd Font 
```
cd ~/Library/Fonts
open . 
# Copy font files to fonts folder
# Caskaydia Cove Nerd Font Complete Regular.otf
# Caskaydia Cove Nerd Font Complete Mono Regular.otf
```

#### coc.nvim 

- Windows pwsh

```powershell
cd <coc.nvim_folder>
sudo nvm use 14.19.3
npm install -g yarn 
yarn install

```


- mac 

```
cd ~/.local/share/nvim/plugged/coc.nvim
yarn install
```<file_sep>+++
title="AWS: Labs"
description="Labs - "
weight=90
draft=true
+++

## Labs 1 

AWS has tons of services available on its resources panel. By reading the document or watching the online tutorial is a simple way to learn key concepts, but only the hands-on approach can guarantee you really master the knowledge and become a known-how AWS expert.

### Prerequisite

* Have an AWS account
* Install AWS Cli tool
* Under how to use AWS Cli
* Most of labs below are tested with AWS Cli

### TODO - Lab 1        

* Create a VPC  - 10.100.0.0/16 
* Create a private subnet and a public subnet
* Create a wp site with mysql db
* Test the wp

```


```


### TODO - Lab 2<file_sep>#![allow(unused_variables)]
fn main() {
    let string1 = String::from("abcd");
    let string2 = "xyz";

    let result = longest(string1.as_str(), string2);
    println!("The longest string is {}", result);
}


fn longest<'a>(x: &'a str, y: &'a str) -> &'a str {
    if x.len() > y.len() {
        x
    } else {
        y
    }
}


// #![allow(unused_variables)]
// fn main() {
//     {
//         let r;

//         {
//             let x = 5;
//             r = &x;
//         }

//         println!("r: {}", r);
//     }
// }


// #![allow(unused_variables)]
// fn main() {
//     enum Coin {
//         Penny,
//         Nickel,
//         Dime,
//         Quarter,
//     }

//     fn value_in_cents(coin: Coin) -> u8 {
//         match coin {
//             Coin::Penny => 1,
//             Coin::Nickel => 5,
//             Coin::Dime => 10,
//             Coin::Quarter => 25,
//         }
//     }
// }


// #[derive(Debug)]
// struct Rectangle {
//     width: u32,
//     height: u32,
// }

// impl Rectangle {
//     fn area(&self) -> u32 {
//         self.width * self.height
//     }

//     fn can_hold(&self, other: &Rectangle) -> bool {
//         self.width > other.width && self.height > other.height
//     }
// }

// fn main() {
//     let rect1 = Rectangle {
//         width: 30,
//         height: 50,
//     };

//     println!(
//         "The area of the rectangle is {} square pixels.",
//         rect1.area()
//     );

//     let rect2 = Rectangle {
//         width: 10,
//         height: 40,
//     };
//     let rect3 = Rectangle {
//         width: 60,
//         height: 45,
//     };

//     println!("Can rect1 hold rect2? {}", rect1.can_hold(&rect2));
//     println!("Can rect1 hold rect3? {}", rect1.can_hold(&rect3));
// }

// fn main() {
//     // let x = 5;

//     // let x = x + 1;

//     // let x = x * 2;

//     // println!("The value of x is: {}", x);

//     // let a = 8;
//     // let b = a;
//     // println!(" a = {} ", a )

//     let s = String::from("hello");

//     let slice1 = &s[0..2];
//     let slice2 = &s[..2];
//     let slice3 = &s[..];

//     println!("{}", slice1 );
//     println!( "{}",slice2);
//     println!( "{}", slice3 );
// }
<file_sep>+++
date = "2018-12-04T14:59:31+11:00"
title = "Amazon Linux 2 "
description = "Amazon Linux 2 - Setup & Configure"
+++

## Amazon Linux 2

Amazon Linux 2 is the next generation of Amazon Linux, a Linux server operating system from Amazon Web Services (AWS). It provides a secure, stable, and high performance execution environment to develop and run cloud and enterprise applications. With Amazon Linux 2, you get an application environment that offers long term support with access to the latest innovations in the Linux ecosystem. Amazon Linux 2 is provided at no additional charge.


### Package update

      sudo yum update

### Get system info

      cat /etc/image-id
      cat /etc/system-release

### Mount a volume (EBS)

```bash
# Get drives info 
suod lsblk
# Get volumen info
sudo file -s /dev/xvdf
# Format volumne
sudo mkfs -t xfs /dev/xvdf
# Mount volume to folder data
sudo mkdir /data
sudo mount /dev/xvdf /data 
```

### Extend the EBS

```bash
sudo xfs_growfs -d /data
```

### Auto attached volume

```bash
sudo cp /etc/fstab /etc/fstab.orig

sudo lsblk -o +UUID
# You will see similar output below
# NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT UUID
# xvda    202:0    0  20G  0 disk
# └─xvda1 202:1    0  20G  0 part /          e75a1891-3463-448b-8f59-5e3353af90ba
# xvdb    202:16   0  60G  0 disk /data      897b5130-c5c1-4ac9-aae3-699d1eaa9fd5

# Use vim to edit /etc/fstab
sudo vim /etc/fstab

# Add following line 
# UUID=897b5130-c5c1-4ac9-aae3-699d1eaa9fd5  /data  xfs defaults,nofail  0  2

# Verify the mounting
sudo umount /data
sudo mount -a

lsblk
```


### Add new user

* Add new user without password

```bash
sudo adduser new_user 

## Add user without password 
sudo adduser new_user  --disabled-password
```

* Switch to new user

```bash
sudo su - new_user
```

* Set password for new user
  
```bash
sudo passwd new_user
```

* Allow the new user to use sudo

```bash
sudo usermod -aG wheel new_user
```

### Install MySql

* Install MySql 5.7

```bash
# Add repo
sudo wget https://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm
sudo yum localinstall mysql57-community-release-el7-11.noarch.rpm 

# Install mysql
sudo yum install mysql-community-server
```

* MySql configuration file sits in /etc/my.cnf

* Update data directory 

```bash
# backup original one
sudo cp /etc/my.cnf /etc/my.cnf.orig
```

* Use vim to update the data directory 

   ```bash
   datadir=/data/mysql
   ```

#### Start MySql as service

* Install polkit before start the service, otherwise you will get error

```bash
sudo yum install polkit
```

* Enable & Start mysql

```bash
sudo systemctl enable mysqld.service
sudo systemctl start mysqld.service
```

* Find the temporay password created for root in /var/log/mysql.log

   ```bash
   sudo cat /var/log/mysql.log | grep "temporary password"
   # output
   # [Note] A temporary password is generated for root@localhost: l<C-eX&GW8?m
   ```

#### Reset root password

      sudo mysql_secure_installation

#### Create remote login credentials

```sql
CREATE USER 'user_id'@'localhost' IDENTIFIED BY 'your_secret';
CREATE USER 'user_id'@'%' IDENTIFIED BY 'your_secret';

GRANT ALL ON *.* TO 'user_id'@'localhost';
GRANT ALL ON *.* TO 'user_id'@'%';
```

### Install AWS CLI 

```bash
# Install aws cli without sudo 
curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
unzip awscli-bundle.zip
./awscli-bundle/install -b ~/bin/aws

# Configure cli 
aws configure
AWS Access Key ID [None]: <KEY>
AWS Secret Access Key [None]: <KEY>
Default region name [None]: region-code
Default output format [None]: json
```

### Install EKS 

```bash
curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin
ekstool --version
```

### Install kubectl

```bash
curl -o kubectl https://amazon-eks.s3.us-west-2.amazonaws.com/1.15.10/2020-02-22/bin/linux/amd64/kubectl
chmod +x ./kubectl
mkdir -p $HOME/bin && cp ./kubectl $HOME/bin/kubectl && export PATH=$PATH:$HOME/bin
```

### Install EPEL repository

```bash
## Amazon Linux 1 / 2
sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo yum update

## Amazon Linux 2
sudo amazon-linux-extras install epel

```







<file_sep>+++
title = "DB & SQL"
description = "RMDB & SQL Notes"
+++

{{%children style="card" description="true" sort="Weight" %}}<file_sep>+++
title="Template"
description="Template"
weight=10
+++

### Template

* Package template implements data-driven templates for generating textual output. 
* Templates are executed by applying them to a data structure. Annotations in the template refer to elements of the data structure (typically a field of a struct or a key in a map) to control execution and derive values to be displayed. Execution of the template walks the structure and sets the cursor, represented by a period '.' and called "dot", to the value at the current location in the structure as execution proceeds.

* The input text for a template is UTF-8-encoded text in any format. "Actions"--data evaluations or control structures--are delimited by "{{" and "}}"; all text outside actions is copied to the output unchanged. Except for raw strings, actions may not span newlines, although comments can. 


#### Pipelines

* A pipeline is a possibly chained sequence of "commands". A command is a simple value (argument) or a function or method call, possibly with multiple arguments.



#### Example of letter code

```go
func main() {
    // Define a template.
    const letter = `
Dear {{.Name}},
{{if .Attended}}
It was a pleasure to see you at the wedding.
{{- else}}
It is a shame you couldn't make it to the wedding.
{{- end}}
{{with .Gift -}}
Thank you for the lovely {{.}}.
{{end}}
Best wishes,
Josie
`

    // Prepare some data to insert into the template.
    type Recipient struct {
        Name, Gift string
        Attended   bool
    }
    var recipients = []Recipient{
        {"<NAME>", "bone china tea set", true},
        {"<NAME>", "moleskin pants", false},
        {"<NAME>", "", false},
    }

    // Create a new template and parse the letter into it.
    t := template.Must(template.New("letter").Parse(letter))

    // Execute the template for each recipient.
    for _, r := range recipients {
        err := t.Execute(os.Stdout, r)
        if err != nil {
            log.Println("executing template:", err)
        }
    }

}

```


#### Glob example

*  demonstrate loading a set of templates from a directory

```go
// templateFile defines the contents of a template to be stored in a file, for testing.
type templateFile struct {
    name     string
    contents string
}

func createTestDir(files []templateFile) string {
    dir, err := ioutil.TempDir("", "template")
    if err != nil {
        log.Fatal(err)
    }
    for _, file := range files {
        f, err := os.Create(filepath.Join(dir, file.name))
        if err != nil {
            log.Fatal(err)
        }
        defer f.Close()
        _, err = io.WriteString(f, file.contents)
        if err != nil {
            log.Fatal(err)
        }
    }
    return dir
}

func main() {
    // Here we create a temporary directory and populate it with our sample
    // template definition files; usually the template files would already
    // exist in some location known to the program.
    dir := createTestDir([]templateFile{
        // T0.tmpl is a plain template file that just invokes T1.
        {"T0.tmpl", `T0 invokes T1: ({{template "T1"}})`},
        // T1.tmpl defines a template, T1 that invokes T2.
        {"T1.tmpl", `{{define "T1"}}T1 invokes T2: ({{template "T2"}}){{end}}`},
        // T2.tmpl defines a template T2.
        {"T2.tmpl", `{{define "T2"}}This is T2{{end}}`},
    })
    // Clean up after the test; another quirk of running as an example.
    defer os.RemoveAll(dir)

    // pattern is the glob pattern used to find all the template files.
    pattern := filepath.Join(dir, "*.tmpl")

    // Here starts the example proper.
    // T0.tmpl is the first name matched, so it becomes the starting template,
    // the value returned by ParseGlob.
    tmpl := template.Must(template.ParseGlob(pattern))

    err := tmpl.Execute(os.Stdout, nil)
    if err != nil {
        log.Fatalf("template execution: %s", err)
    }
}
```


#### Helper example

* demonstrates one way to share some templates and use them in different contexts. In this variant we add multiple driver templates by hand to an existing bundle of templates.

```go

// templateFile defines the contents of a template to be stored in a file, for testing.
type templateFile struct {
    name     string
    contents string
}

func createTestDir(files []templateFile) string {
    dir, err := ioutil.TempDir("", "template")
    if err != nil {
        log.Fatal(err)
    }
    for _, file := range files {
        f, err := os.Create(filepath.Join(dir, file.name))
        if err != nil {
            log.Fatal(err)
        }
        defer f.Close()
        _, err = io.WriteString(f, file.contents)
        if err != nil {
            log.Fatal(err)
        }
    }
    return dir
}

func main() {
    // Here we create a temporary directory and populate it with our sample
    // template definition files; usually the template files would already
    // exist in some location known to the program.
    dir := createTestDir([]templateFile{
        // T1.tmpl defines a template, T1 that invokes T2.
        {"T1.tmpl", `{{define "T1"}}T1 invokes T2: ({{template "T2"}}){{end}}`},
        // T2.tmpl defines a template T2.
        {"T2.tmpl", `{{define "T2"}}This is T2{{end}}`},
    })
    // Clean up after the test; another quirk of running as an example.
    defer os.RemoveAll(dir)

    // pattern is the glob pattern used to find all the template files.
    pattern := filepath.Join(dir, "*.tmpl")

    // Here starts the example proper.
    // Load the helpers.
    templates := template.Must(template.ParseGlob(pattern))
    // Add one driver template to the bunch; we do this with an explicit template definition.
    _, err := templates.Parse("{{define `driver1`}}Driver 1 calls T1: ({{template `T1`}})\n{{end}}")
    if err != nil {
        log.Fatal("parsing driver1: ", err)
    }
    // Add another driver template.
    _, err = templates.Parse("{{define `driver2`}}Driver 2 calls T2: ({{template `T2`}})\n{{end}}")
    if err != nil {
        log.Fatal("parsing driver2: ", err)
    }
    // We load all the templates before execution. This package does not require
    // that behavior but html/template's escaping does, so it's a good habit.
    err = templates.ExecuteTemplate(os.Stdout, "driver1", nil)
    if err != nil {
        log.Fatalf("driver1 execution: %s", err)
    }
    err = templates.ExecuteTemplate(os.Stdout, "driver2", nil)
    if err != nil {
        log.Fatalf("driver2 execution: %s", err)
    }
}

```<file_sep>+++
title = "Shell"
description = "Shell, Batch & Powershell Notes"
+++

{{%children style="card" description="true" sort="Weight" %}}<file_sep>+++

date = "2016-12-07T16:56:21+11:00"
title = "Vue 2 Admin Project"

description="Vue2Admin is a fully responsive admin template that is inspired by AdminLTE"
+++

## Summary

**Vue2Admin** is a fully responsive admin template that is inspired by [AdminLTE](https://almsaeedstudio.com). 

## Features

* This template is built-in with Vue 2 at the start.
* It include Vue Resource, Vuex as well.
* The plugins for this project are loaded with script loader. 

## Structure of Reetek Vue2Admin

``` ini
path\to\vue2admin
+---build              <-// webpack files
+---config             <-// configuration of dev or prod environment
+---screenshots
+---src                <-// vue components 
|   +---components
|   |   +---charts
|   |   +---dashboard
|   |   +---forms
|   |   +---mailbox
|   |   +---misc
|   |   +---pages
|   |   +---tables
|   |   +---ui
|   |   \---widget
|   \---filters
+---static              <-// css, dump data, fonts, image files and plugins
|   +---css
|   +---data
|   +---fonts
|   +---img
|   |   +---credit
|   |   \---stock
|   \---js
|       \---plugins         <-// plugins for admin dashboard 
|           +---AdminLTE
|           +---bootstrap
|           +---bootstrap-slider
|           +---bootstrap-wysihtml5
|           +---chartjs
|           +---ckeditor
|           + ...
\---test
    +---e2e
    \---unit
```


## Screenshots

> ![Screenshot1](/img/vue2admin_screenshot1_sm.png)

> ![Screenshot2](/img/vue2admin_screenshot2_sm.png)

## Browse [Repository](https://github.com/harryho/vue2admin.git)
<file_sep>+++
title = "AWS: Mesh - 1"
description = "Getting started"
draft="true"
+++


## Mesh 

AWS App Mesh is a service mesh based on the Envoy proxy that helps you monitor and control services. App Mesh standardizes how your services communicate, giving you end-to-end visibility into and helping to ensure high-availability for your applications. 



#### Create a mesh and virtual service

A service mesh is a logical boundary for network traffic between the services that reside within it.


  - A mesh named pgMesh, since all of the services in the scenario are registered to the pgMesh.local namespace.

  - A virtual service named svc1.pgMesh.local, since the virtual service represents a service that is discoverable with that name, and you don't want to change your code to reference another name. A virtual service named svc2.pgMesh.local is added in a later step.

        aws appmesh create-mesh --mesh-name pgMesh


  - Create a virtual service with the create-virtual-service command.

        aws appmesh create-virtual-service \
                --mesh-name pgMesh \
                --virtual-service-name svc1.pgMesh.local \
                --spec {}

#### Create a virtual node
  
A virtual node acts as a logical pointer to an actual service. 

Create a virtual node named svcNode1, since one of the virtual nodes represents the actual service named svcNode1. The actual service that the virtual node represents is discoverable through DNS with a hostname of svc1.pgMesh.local. Alternately, you can discover actual services using AWS Cloud Map. The virtual node will listen for traffic using the HTTP/2 protocol on port 80. 


* Create a file named create-virtual-node-svc1.json with the following contents:

```json
{
    "meshName": "pgMesh",
    "spec": {
        "listeners": [
            {
                "portMapping": {
                    "port": 80,
                    "protocol": "http2"
                }
            }
        ],
        "serviceDiscovery": {
            "dns": {
                "hostname": "svcNode1.pgMesh.local"
            }
        }
    },
    "virtualNodeName": "svcNode1"
}
```

* Create the virtual node with the create-virtual-node command using the JSON file as input.

        aws appmesh create-virtual-node --cli-input-json file://create-virtual-node-svc1.json


  
#### Create a virtual router and route
  
Virtual routers route traffic for one or more virtual services within your mesh. 

Create the following resources:

> A virtual router named svcNode1, since the svcNode1.pgMesh.local virtual service doesn't initiate outbound communication with any other service. Remember that the virtual service that you created previously is an abstraction of your actual svc1.pgMesh.local service. The virtual service sends traffic to the virtual router. The virtual router will listen for traffic using the HTTP/2 protocol on port 80. Other protocols are also supported.

> A route named svcNode1. It will route 100 percent of its traffic to the svcNode1 virtual node. You'll change the weight in a later step once you've added the svcNode1v2 virtual node. Though not covered in this guide, you can add additional filter criteria for the route and add a retry policy to cause the Envoy proxy to make multiple attempts to send traffic to a virtual node when it experiences a communication problem.


#### Create a virtual router

Create a file named create-virtual-router.json with the following contents:

```json
{
    "meshName": "pgMesh",
    "spec": {
        "listeners": [
            {
                "portMapping": {
                    "port": 80,
                    "protocol": "http2"
                }
            }
        ]
    },
    "virtualRouterName": "svcNode1"
}
```


* Create the virtual router with the create-virtual-router command using the JSON file as input.

        aws appmesh create-virtual-router --cli-input-json file://create-virtual-router.json

* Create a route.

  - Create a file named create-route.json with the following contents:

```json
{
    "meshName" : "pgMesh",
    "routeName" : "svcNode1",
    "spec" : {
        "httpRoute" : {
            "action" : {
                "weightedTargets" : [
                    {
                        "virtualNode" : "svcNode1",
                        "weight" : 100
                    }
                ]
            },
            "match" : {
                "prefix" : "/"
            }
        }
    },
    "virtualRouterName" : "svcNode1"
}
```

* Create the route with the create-route command using the JSON file as input.

        aws appmesh create-route --cli-input-json file://create-route.json


#### Review and create

Review the settings against the previous instructions.

* Review the settings of the mesh you created with the describe-mesh command.

        aws appmesh describe-mesh --mesh-name pgMesh

* Review the settings of the virtual service that you created with the describe-virtual-service command.

        aws appmesh describe-virtual-service --mesh-name pgMesh --virtual-service-name svc1.pgMesh.local

* Review the settings of the virtual node that you created with the describe-virtual-node command.

        aws appmesh describe-virtual-node --mesh-name pgMesh --virtual-node-name svcNode1

* Review the settings of the virtual router that you created with the describe-virtual-router command.

        aws appmesh describe-virtual-router --mesh-name pgMesh --virtual-router-name svcNode1

* Review the settings of the route that you created with the describe-route command.

        aws appmesh describe-route --mesh-name pgMesh \
                --virtual-router-name svcNode1  --route-name svcNode1


#### Create additional resources

Create one virtual node named svcNode1v2 and another named svcNode2. Both virtual nodes listen for requests over HTTP/2 port 80. For the svcNode2 virtual node, configure a backend of svc1.pgMesh.local, since all outbound traffic from the svcNode2 virtual node is sent to the virtual service named svc1.pgMesh.local. Though not covered in this guide, you can also specify a file path to write access logs to for a virtual node.

Create one additional virtual service named svc2.pgMesh.local, which will send all traffic directly to the svcNode2 virtual node.

Update the svcNode1 route that you created in a previous step to send 75 percent of its traffic to the svcNode1 virtual node and 25 percent of its traffic to the svcNode1v2 virtual node. Over time, you can continue to modify the weights until svcNode1v2 receives 100 percent of the traffic. 

Once all traffic is sent to svcNode1v2, you can deprecate the svcNode1 virtual node and actual service. As you change weights, your code doesn't require any modification, because the svc1.pgMesh.local virtual and actual service names don't change. Recall that the svc1.pgMesh.local virtual service sends traffic to the virtual router, which routes the traffic to the virtual nodes. The service discovery names for the virtual nodes can be changed at any time.

* Create the svcNode1v2 virtual node.

* Create a file named create-virtual-node-svcnode1v2.json with the following contents:

```json
{
    "meshName": "pgMesh",
    "spec": {
        "listeners": [
            {
                "portMapping": {
                    "port": 80,
                    "protocol": "http2"
                }
            }
        ],
        "serviceDiscovery": {
            "dns": {
                "hostname": "svcNode1v2.pgMesh.local"
            }
        }
    },
    "virtualNodeName": "svcNode1v2"
}
```


* Create the virtual node.

        aws appmesh create-virtual-node --cli-input-json file://create-virtual-node-svcnode1v2.json

* Create the svcNode2 virtual node.

* Create a file named create-virtual-node-svc2.json with the following contents:

```json
{
   "meshName" : "pgMesh",
   "spec" : {
      "backends" : [
         {
            "virtualService" : {
               "virtualServiceName" : "svc1.pgMesh.local"
            }
         }
      ],
      "listeners" : [
         {
            "portMapping" : {
               "port" : 80,
               "protocol" : "http2"
            }
         }
      ],
      "serviceDiscovery" : {
         "dns" : {
            "hostname" : "svc2.pgMesh.local"
         }
      }
   },
   "virtualNodeName" : "svcNode2"
}

```


* Create the virtual node


        aws appmesh create-virtual-node --cli-input-json file://create-virtual-node-svc2.json


Update the svc1.pgMesh.local virtual service that you created in a previous step to send its traffic to the svcNode1 virtual router. When the virtual service was originally created, it didn't send traffic anywhere, since the svcNode1 virtual router hadn't been created yet.

Create a file named update-virtual-service.json with the following contents:

```json
{
   "meshName" : "pgMesh",
   "spec" : {
      "provider" : {
         "virtualRouter" : {
            "virtualRouterName" : "svcNode1"
         }
      }
   },
   "virtualServiceName" : "svc1.pgMesh.local"
}
```


* Update the virtual service with the update-virtual-service command.

        aws appmesh update-virtual-service --cli-input-json file://update-virtual-service.json

* Update the svcNode1 route that you created in a previous step.

  - Create a file named update-route.json with the following contents:

```json
{
   "meshName" : "pgMesh",
   "routeName" : "svcNode1",
   "spec" : {
      "http2Route" : {
         "action" : {
            "weightedTargets" : [
               {
                  "virtualNode" : "svcNode1",
                  "weight" : 75
               },
               {
                  "virtualNode" : "svcNode1v2",
                  "weight" : 25
               }
            ]
         },
         "match" : {
            "prefix" : "/"
         }
      }
   },
   "virtualRouterName" : "svcNode1"
}
```

* Update the route with the update-route command.

        aws appmesh update-route --cli-input-json file://update-route.json

* Create the svcNode2 virtual service.

  - Create a file named create-virtual-svc2.json with the following contents:


```json
{
   "meshName" : "pgMesh",
   "spec" : {
      "provider" : {
         "virtualNode" : {
            "virtualNodeName" : "svcNode2"
         }
      }
   },
   "virtualServiceName" : "svc2.pgMesh.local"
}
```

* Create the virtual service.

        aws appmesh create-virtual-service --cli-input-json file://create-virtual-svc2.json


#### Mesh summary


> Before you created the service mesh, you had three actual services named svc2.pgMesh.local, svc1.pgMesh.local, and servicebv2.pgMesh.local. In addition to the actual services, you now have a service mesh that contains the following resources that represent the actual services:

> Two virtual services. The proxy sends all traffic from the svc2.pgMesh.local virtual service to the svc1.pgMesh.local virtual service through a virtual router.

> Three virtual nodes named svcNode2, svcNode1, and svcNode1v2. The Envoy proxy uses the service discovery information configured for the virtual nodes to look up the IP addresses of the actual services.

> One virtual router with one route that instructs the Envoy proxy to route 75 percent of inbound traffic to the svcNode1 virtual node and 25 percent of the traffic to the svcNode1v2 virtual node.

<file_sep>+++

title = "Powershell Note - 1"
description = "Introduction of Powershell "
+++

> PowerShell is a task-based command-line shell and scripting language built on .NET. PowerShell helps system administrators and power-users rapidly automate tasks that manage operating systems (Linux, macOS, and Windows) and processes.

## Prerequisites

* The OS of Windows 7 or later version
* Install Powershell 4 or later version. You can find it on [Microsoft](www.microsoft.com) website
* Has basic computer knowledge

## Launch PS command prompt 

* Type command on windows command prompt: **powershell**


### Get PS Version

* Type **$psversiontable**



```
    PS C:\>$psversiontable
    ## You might see sth below

    Name                           Value
    ----                           -----
    PSVersion                      4.0
    WSManStackVersion              3.0
    SerializationVersion           1.1.0.1
    CLRVersion                     4.0.30319.42000
    BuildVersion                   6.3.9600.18773
    PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0}
    PSRemotingProtocolVersion      2.2
```


### Install & Uninstall service

```ps
# Install service
New-Service -Name "Your_Service_Name" -BinaryPathName "C:\path_to_your_service\your_service.exe -k netsvcs"

# Uninstall service 
(Get-WmiObject -Class Win32_Service -Filter "Name='Your_Service_Name'").delete()
```


### Create new login & pass

```ps

$Username = 'domain\username'
$PassTxt = '<PASSWORD>'
$Password = ConvertTo-SecureString -AsPlainText $PassTxt -Force
 set-executionpolicy remotesigned; 
New-LocalUser $Username  -Password $Password -FullName $Username -Description $Username
Add-LocalGroupMember -Group "Administrators" -Member $Username
Add-LocalGroupMember -Group "Remote Desktop Users" -Member $Username

```



<file_sep>+++
title = "AWS: EKS - 2"
description = "Update / Upgrade Kubernetes"
weight=12
+++


## EKS - Part 2

The update process consists of Amazon EKS launching new API server nodes with the updated Kubernetes version to replace the existing ones. Amazon EKS performs standard infrastructure and readiness health checks for network traffic on these new nodes to verify that they are working as expected. If any of these checks fail, Amazon EKS reverts the infrastructure deployment, and your cluster remains on the prior Kubernetes version. Running applications are not affected, and your cluster is never left in a non-deterministic or unrecoverable state. Amazon EKS regularly backs up all managed clusters, and mechanisms exist to recover clusters if necessary. We are constantly evaluating and improving our Kubernetes infrastructure management processes.

### Kubernete Info

* Get cluster & context info
        
        kubectl config get-clusters
        kubectl config use-context <context-name>

* Get kubernete version 

        kubectl version --short

* Get nodes info

        kubectl get nodes

* Get pod securtiy policy 

        kubectl get psp eks.privileged

* Get DNS controller info

        kubectl describe deployment coredns --namespace kube-system | grep Image | cut -d "/" -f 3


### Update Kubernete

    eksctl update cluster --name <cluster-name> --approve


### VPC CNI

* Get VPC CNI version

        kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2

* Patch VPC CNI to latest version

        kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/release-1.5/config/v1.5/aws-k8s-cni.yaml


### Cluster Endpoint

* Enable private access for specific IP


        CIDR="192.168.3.11"
        CLUSTER_NAME="pg-prd"
        REGION_CODE="ap-southeast-2"
        aws eks update-cluster-config \
        --region ${REGION_CODE} \
        --name ${CLUSTER_NAME} \
        --resources-vpc-config endpointPublicAccess=true,publicAccessCidrs="${CIDR}/32",endpointPrivateAccess=true


* Check the update status with update-id from above output

        aws eks describe-update \
            --region  ${REGION_CODE} \
            --name ${CLUSTER_NAME} \
            --update-id <update-id>


### Control Plane Logs


* Enable logging

        CLUSTER_NAME="pg-prd"
        REGION_CODE="ap-southeast-2"
        aws eks --region ${REGION_CODE} \
        update-cluster-config --name ${CLUSTER_NAME} \
        --logging '{"clusterLogging":[{"types":["api","audit","authenticator","controllerManager","scheduler"],"enabled":true}]}'

* Check the update status

        aws eks describe-update \
            --region  ${REGION_CODE} \
            --name ${CLUSTER_NAME} \
            --update-id <update-id><file_sep>+++
date = "2011-03-09T10:59:31+11:00"
title = "CMMI"
description="Comparison between Scrum  & Kanban"
weight=1
+++

## CMMI

The Capability Maturity Model Integration (CMMI) is a process and behavioral model that helps organizations streamline process improvement and encourage productive, efficient behaviors that decrease risks in software, product and service development.

### CMMI model

The CMMI starts with an appraisal process that evaluates three specific areas: process and service development, service establishment and management, and product and service acquisition. It’s designed to help improve performance by providing businesses with everything they need to consistently develop better products and services.

But the CMMI is more than a process model; it’s also a behavioral model. Businesses can use the CMMI to tackle the logistics of improving performance by developing measurable benchmarks, but it can also create a structure for encouraging productive, efficient behavior throughout the organization.


### CMMI Maturity Levels

The CMMI model breaks down organizational maturity into five levels. 

- Initial: Processes are viewed as unpredictable and reactive. At this stage, “work gets completed but it’s often delayed and over budget.” This is the worst stage a business can find itself in — an unpredictable environment that increases risk and inefficiency.

- Managed: There’s a level of project management achieved. Projects are “planned, performed, measured and controlled” at this level, but there are still a lot of issues to address.

- Defined: At this stage, organizations are more proactive than reactive. There’s a set of “organization-wide standards” to “provide guidance across projects, programs and portfolios.” Businesses understand their shortcomings, how to address them and what the goal is for improvement.

- Quantitatively managed: This stage is more measured and controlled. The organization is working off quantitative data to determine predictable processes that align with stakeholder needs. The business is ahead of risks, with more data-driven insight into process deficiencies.

- Optimizing: Here, an organization’s processes are stable and flexible. At this final stage, an organization will be in constant state of improving and responding to changes or other opportunities. The organization is stable, which allows for more “agility and innovation,” in a predictable environment.

Once organizations hit Levels 4 and 5, they are considered high maturity, where they are “continuously evolving, adapting and growing to meet the needs of stakeholders and customers.” That is the goal of the CMMI: To create reliable environments, where products, services and departments are proactive, efficient and productive.


<file_sep>+++
title = "Java Note - 5: Lambda "
description="Lambda expressions are Java's first step into functional programming"
+++


## Lamda

### Lambda Best Practices

## Use Interfaces
The most common misstep taken by an over-eager functional programmer is the use of functional interfaces
in type signatures. In general, you should avoid using the functional interface types directly and instead
provide single-method interfaces as arguments to your methods. These interfaces become a way to create
self-documenting code and to provide meaningful type information, as well as leaving open the opportunity
for your user to provide an actual Java type. 

## Use Method Reference
As much as possible, use a method reference instead of a lambda. Method references are not only shorter
and easier to read, but using method references will get you thinking directly about the methods as values.


## Define Lambdas Inline
When you do use lambdas, define them inline. Unless you are doing some kind of fancy manipulation of your
lambda, there is no reason to be assigning them to a variable. The reason that you want to define your lambdas
inline is that it will allow your code to be more flexible when types change: you are letting type inference do
more heavy lifting for you, and adapting your code to changing contexts. 

## Lambdas Should Always Be Threadsafe
As we go through the rest of this book, we will see many places where lambdas make concurrent programming
much easier. Many of the structures built off of lambdas will perform concurrent executions, sometimes without
much warning. Because of this, your lambdas always need to be threadsafe. Pay particular attention to this with
instance method handles, since thread-dangerous state can often be hiding within those instances.

## Don’t Use Null
The null keyword should never be used in your code. Now that Java has the Optional type, there is simply
no need for it. Whenever you have a method, you should be explicit about whether or not you accept
null, and you generally shouldn’t accept it. This will save you from NullPointerException cropping up
in obnoxious places, far from the site of the actual error. This is an especially painful problem when you
start working with streams and lambdas, because the stack trace may not be very useful for you when
you go to debug. The solution is to never accept null and to aggressively check for it, exploding loudly as
soon as it occurs. 

## Don’t Release Zalgo
* Don't mix asynchronous and synchronous execution in the same lamda expressoin.

## Build Complexity from Simple Parts

## Use Types and the Compiler to Your Advantage


### Common functional Interfaces

|Functional Interface|Parameter Types|Return|Abstract Description Method|Description| Other Methods |
|----------|-----------|------|----------------|-----------|-----|
|Runnable|none|void|run|Runs an action|  |
|Supplier<T>|none|T|get|Supplies a value of type T|  |
|Consumer<T>|T|void|accept|Consumes a  value of type T| chain  |
|BiConsumer<T, U>|T, U|void|accept|Consumes a  value of type T and U| chain  |
|Function<T, R>| T |R | apply| A function with argument oftype T|compose, andThen,identity|
|BiFunction<T, U, R>| T, U |R | apply| A function with argument of type T and U|andThen|
|UnaryOperator<T>| T |T | apply| A unary operator on type T|compose, andThen, identity|
|BiUnaryOperator<T,T>| T,T |T | apply| A binary operator on type T|andThen|
|Predicate<T>| T |boolean | test| A Boolean-valued function |and,  or, negate, isEqual|
|BiPredicate<T,T>| T,T |boolean | test| A Boolean-valued function with tow arguments|and,  or, negate|


### Method Reference

| Syntax | Description |
|--------|-------------|
| TypeName::staticMethod | A method reference to a static method of a class, an interface, or an enum |
|objectRef::instanceMethod |A method reference to an instance method of the specified object|
|ClassName::instanceMethod |A method reference to an instance method of an arbitrary object of the specified class|
|TypeName.super::instanceMethod |A method reference to an instance method of the supertype of a particular object |
|ClassName::new | A constructor reference to the constructor of the specified class |
|ArrayTypeName::new | An array constructor reference to the constructor of the specified
array type |


### Lambda Demo

<!-- <iframe height="800px" width="100%" src="https://repl.it/@harryh0/LambdaDemo?lite=true" scrolling="no" frameborder="no" allowtransparency="true" allowfullscreen="true" sandbox="allow-forms allow-pointer-lock allow-popups allow-same-origin allow-scripts allow-modals"></iframe>

[lambda-demo](https://repl.it/@harryh0/AttentiveElementaryTriangles) -->

```java
import java.util.Locale;
import java.util.Arrays;
import java.util.List;
import java.util.ArrayList;
import java.util.function.Supplier;
import java.util.function.Consumer;
import java.util.function.Function;
import java.util.function.BiFunction;
import java.util.function.Predicate;
import java.util.function.UnaryOperator;
import java.util.function.BinaryOperator;
import java.util.function.IntFunction;

public class LambdaDemo {

    public static void main(String[] args) {

        // FunctionalInterface 
        System.out.println("x + y:" + engine((x, y) -> x + y)); // 6
        System.out.println("x * y:" + engine((x, y) -> x * y)); // 8
        System.out.println("x / y:" + engine((x, y) -> x / y)); // 0 
        System.out.println("x % y:" + engine((x, y) -> x % y)); // 2

        String[] strArray = new String[] { "abc", "klm", "xyz", "pqr" };
        List list = Arrays.asList(strArray);

        // Default Methods
        list.forEach(System.out::println);

        // abc
        // klm
        // xyz
        // pqr

        Arrays.sort(strArray, (first, second) -> first.compareToIgnoreCase(second));
        list = Arrays.asList(strArray);

        System.out.println("After sorting ... ");
        list.forEach(System.out::println);

        // After sorting ...
        // abc
        // klm
        // pqr
        // xyz

        // Common Functional Interfaces
        // Runnable
        repeat(5, () -> System.out.println("Hello"));

        UnaryOperator<String> upperCase = str -> str.toUpperCase();
        BinaryOperator<String> concat = (left, right) -> left + right;

        System.out.println(" UnaryOperator upperCase " + upperCase.apply("hello"));
        System.out.println("  BinaryOperator<String> concat " + concat.apply("hello", "world"));

        // Function 
        Function<Long, Long> square = x -> x * x;
        Function<Long, Long> plusOne = x -> x + 1;
        // Function with andThen, 
        Function<Long, Long> squarePlusOne = square.andThen(plusOne);
        Function<Long, Long> plusOneSquare = square.compose(plusOne);
        System.out.println(" 5 squarePlusOne is " + squarePlusOne.apply(5L)); // 26 
        System.out.println(" 5 plusOneSquare is  " + plusOneSquare.apply(5L)); // 36

        // Predicate
        Predicate<Integer> divisibleByThree = x -> x % 3 == 0;
        Predicate<Integer> divisibleByFive = x -> x % 5 == 0;
        Predicate<Integer> isNegative = x -> x < 0;
        // Predicate with AND , OR , NOT
        Predicate<Integer> divisibleByThreeAndFive = divisibleByThree.and(divisibleByFive);
        Predicate<Integer> divisibleByThreeOrFive = divisibleByThree.or(divisibleByFive);
        Predicate<Integer> isPositive = isNegative.negate();

        System.out.println(" 15 is divisibleByThreeAndFive " + divisibleByThreeAndFive.test(15));
        System.out.println(" 7 is divisibleByThreeAndFive " + divisibleByThreeOrFive.test(7));
        System.out.println(" -1 is isPositive " + isPositive.test(7));

        // static method reference 
        Function<Integer, String> toBinary = x -> Integer.toBinaryString(x);
        System.out.println(toBinary.apply(19));

        // Using a method reference
        Function<Integer, String> toBinary2 = Integer::toBinaryString;
        System.out.println(toBinary2.apply(19));

        // static method lambda expression
        BiFunction<Integer, Integer, Integer> sum = (a, b) -> Integer.sum(a, b);
        System.out.println(sum.apply(3, 4));

        // Instance method
        Supplier<Person> personSup = () -> new Person();
        Function<String, Person> personFunc = (x) -> new Person(x);

        BiFunction<String, String, Person> personBiFunc = (x, y) -> new Person(x, y);
        // Consumer<String> personCon = (Person p) -> p.setTitle;

        System.out.println(personSup.get());
        // Person() constructor called
        // name = Unknown, title = Unknown
        System.out.println(personFunc.apply("<NAME>"));
        // Person( fullName ) constructor called
        // name = <NAME>, title = Unknown

        System.out.println(personBiFunc.apply("John", "Doe"));
        // Person(firstName, lastName ) constructor called
        // name = John, title = Unknown


        // Recursive Lambda Expressions
        IntFunction<Long> factorialCalc = new IntFunction<Long>() {
            @Override
            public Long apply(int n) {
                if (n < 0) {
                    String msg = "Number must not be negative.";
                    throw new IllegalArgumentException(msg);
                }
                if (n == 0) {
                    return 1L;
                } else {
                    return n * this.apply(n - 1);
                }
            }
        };

        int n = 5;
        long fact = factorialCalc.apply(n);
        System.out.println("Factorial of " + n + " is " + fact);
        // Factorial of 5 is 120   
    }

    private static int engine(Calculator calculator) {
        int x = 2, y = 4;
        return calculator.calculate(x, y);
    }

    public static void repeat(int n, Runnable action) {
        for (int i = 0; i < n; i++)
            action.run();
    }
}

@FunctionalInterface
interface Calculator {
    int calculate(int x, int y);
}

final class Person {
    String firstName;
    String lastName;
    String fullName;
    String title;

    public Person() {
        System.out.println(" Person() constructor called ");
    }

    public Person(String fullName) {
        this.fullName = fullName;
        System.out.println(" Person( fullName ) constructor called ");
    }

    public Person(String firstName, String lastName) {
        this.firstName = firstName;
        this.lastName = lastName;
        System.out.println(" Person(firstName, lastName ) constructor called ");
    }

    public void setTitle(String t) {
        this.title = t;
        System.out.println(" Person setTitle ( t ) called ");
    }

    public String getFirstName() {
        return firstName;
    }

    public String getFullName() {
        return fullName == null ?( firstName != null ? firstName : "Unknown" ): fullName;
    }

    @Override
    public String toString() {
        return "name = " + getFullName() + ", title = " + (title != null ? title : "Unknown");
    }
}


```

<file_sep>+++


date = "2016-04-10T14:59:31+11:00"
title = "Python & MoviePy"
draft = true
+++

## Python

### Prerequisites 

* Windows based development environment. 

### Install Anaconda

### Install MoviePy
`\path\to\andacoda\scripts\pip install -i https://pypi.anaconda.org/pypi/simple moviepy`

### Download FFMpeg & ImageMagicK
* Download the lasted [static file](https://ffmpeg.zeranoe.com/builds/win64/static/). You can choose zip or 7z format, and then extra to your local PC. 
* Download the [ImageMagick](https://www.imagemagick.org/script/download.php) from its website. I will suggest download ImageMagick-7.0.5-0-portable-Q16-x86.zip (32bit) or ImageMagick-x.x.x.x-portable-Q16-x64.zip (64bit) and extra it to local PC. 
* Update the `config.py` within the `moviepy` module to setup the correct paths for `FF`

### Setup FFMpeg

```

```

### FFMpeg manipulation
```
<file_sep>+++
title = "AWS: ECS - 2"
description = "Getting started with EC2"
draft="true"
+++


## ECS - 2

As mentioned before, you can host your tasks on a cluster of Amazon Elastic Compute Cloud (Amazon EC2) instances that you manage by using the EC2 launch type. 


###  Launcth with EC2 


* Create a cluster configuration
  
        ecs-cli configure --cluster pg-ec2 \
                --default-launch-type EC2 \
                --config-name pg-ec2 \
                --region ap-southeast-2


* Create a CLI profile 

        ecs-cli configure profile \
            --access-key AWS_ACCESS_KEY_ID \
            --secret-key AWS_SECRET_ACCESS_KEY \
            --profile-name pg-ec2-profile



* Create a Cluster 
  
        ecs-cli up --keypair id_rsa \
            --capability-iam --size 2 \
            --instance-type t2.medium \
            --cluster-config ec2-tutorial \
            --ecs-profile ec2-tutorial-profile

* Create a Compose File - docker-compose.yml

```yaml
version: '3'
services:
web:
    image: amazon/amazon-ecs-sample
    ports:
    - "80:80"
    logging:
    driver: awslogs
    options: 
        awslogs-group: pg-far
        awslogs-region: ap-southeast-2
        awslogs-stream-prefix: web
```

* Create a parameters specific file - ecs-params.yml


```yaml
version: 1
task_definition:
  services:
    web:
      cpu_shares: 100
      mem_limit: 524288000
```

* Deploy the Compose File to a Cluster

        ecs-cli compose up --create-log-groups \
                --cluster-config ec2-tutorial \
                --ecs-profile ec2-tutorial-profile

* View the container

        ecs-cli ps --cluster-config ec2-tutorial \
                --ecs-profile ec2-tutorial-profile

* Scale the Tasks on a Cluster

        ecs-cli compose scale 2 \
                --cluster-config ec2-tutorial \
                --ecs-profile ec2-tutorial-profile

* View the container

        ecs-cli ps --cluster-config ec2-tutorial \
                --ecs-profile ec2-tutorial-profile

* Create an ECS Service

        # Stop the containers first
        ecs-cli compose down \
                --cluster-config ec2-tutorial \
                --ecs-profile ec2-tutorial-profile

        # Create the service.
        ecs-cli compose service up \
                --cluster-config ec2-tutorial \
                --ecs-profile ec2-tutorial-profile

<file_sep>+++
title = "C#"
description = "C# Tutorials"
+++

{{%children style="card" description="true" sort="Weight" %}}<file_sep>+++
title = "Azure"
weight = 3
+++

{{%children style="card" description="true" sort="Weight" %}}<file_sep>+++
title ="JS & TS "
description="JavaScript & TypeScript"
+++

{{%children style="card" description="true" sort="Weight" %}}<file_sep>+++

title = "C Lecture - 1"
description = "Exercise 0 ~ 31"
+++

Author: <NAME>

All content comes from Zed's [Lecture Repository](https://github.com/zedshaw/learn-c-the-hard-way-lectures.git) and [Libraries Repository](https://github.com/zedshaw/liblcthw).  All credit goes to Zed.




### Exercise 0 Installing Software

The Plan

* Install software on your system.
* Test that it works right.

Linux Install

On Debian/Ubuntu use:

    $ sudo apt-get install build-essential

On RedHat/CentOS:

    $ sudo yum groupinstall development-tools

Linux Testing

Test that your C compiler works with:

    $ cc --version

OSX Install

Install [XCode](https://developer.apple.com/xcode/), this will take a while.

OSX Testing

Test that your C compiler works with:

    $ cc --version

Windows Install

Install [MinGW](http://www.mingw.org/) or [Cygwin](https://www.cygwin.com/) or Use [VirtualBox](https://www.virtualbox.org/wiki/Downloads) to run Linux.

Text Editors

You should already have one.
Just don't use an IDE. They aren't very helpful.

End of Lecture 0

### Exercise 1 Dust Off That Compiler

The Plan

* Write your first C program.
* Build it.
* Break it.

The Code

.\ex01\ex1.c

```c
#include <stdio.h>

/* This is a comment. */
int main(int argc, char *argv[])
{
    int distance = 100;

    // this is also a comment
    printf("You are %d miles away.\n", distance);

    return 0;
}

```

.\ex01\ex1_zed.c

```c
#include <stdio.h>

/* This is a comment. */
int main(int argc, char *argv[])
{
    int distance = 100;

    // this is also a comment
    printf("You are %d miles away.\n");

    return 0;
}

```

The Analysis

Let's look at it line-by-line.

Breaking It

This is all crazy magic right now.

Extra Credit

* Open the ``ex1`` file in your text editor and change or delete random parts.
  Try running it and see what happens.
* Print out five more lines of text or something more complex than "hello world."
* Run ``man 3 printf`` and read about this function and many others.
* For each line, write out the symbols you don't understand and
  see if you can guess what they mean.  Write a little chart on
  paper with your guess so you can check it later to see
  if you got it right.

### Exercise 2 Using Makefiles to Build

The Plan

* Start with simple make usage.
* Set a few important settings.

How Make Works

Implied dependencies and ancient lore.

Shell Commands

    $ make ex1
    ## or this one too
    $ CFLAGS="-Wall" make ex1
    $ make clean
    $ make ex1

Makefile

    CFLAGS=-Wall -g

    clean:
        rm -f ex1

The Analysis

* Setting options.
* Indicating dependencies.
* Writing commends to run.

Breaking It

* Watch out for tabs vs. spaces.

Extra Credit

* Create an *all: ex1* target that will build *ex1* with
  just the command *make*.
* Read *man make* to find out more information on how to run it.
* Read *man cc* to find out more information on what the flags *-Wall* and *-g* do.
* Research *Makefiles* online and see if you can improve this one.
* Find a *Makefile* in another C project and try to understand
  what it's doing.

### Exercise 3 Formatted Printing

The Plan

* Introduction to *printf*.

The Code

.\ex03\ex3.c

```c
#include <stdio.h>

int main(int argc, char *argv[])
{
    int age = 100;
    int height = 72;

    printf("I am %d years old.\n", argv);
    printf("I am %d inches tall.\n", height);

    return 0;
}

```

The Analysis

Breaking It

* Take the *age* variable out of the first *printf* call, then recompile. You should get a couple of warnings.
* Run this new program and it will either crash or print out a really crazy age.
* Put the *printf* back the way it was, and then don't set *age* to an initial value by changing that line to *int age;*, and then rebuild it and run it again.

Extra Credit

* Find as many other ways to break *ex3.c* as you can.
* Run *man 3 printf* and read about the other *%* format
  characters you can use.  These should look familiar if you used
  them in other languages (they come from *printf*).
* Add *ex3* the *all* list in your *Makefile*.  Use this
  to *make clean all* and build all of your exercises thus far.
* Add *ex3* to your *clean* list in your*Makefile* as well.
  Use *make clean* to remove it when you need to.

### Exercise 4 Using a Debugger

The Plan

* See how GDB works (LLDB on OSX).
* Look at memory checkers like Valgrind and AddressSanitizer.
* Cover the quick reference.
* Debug a program.

Using GDB

Using LLDB

Using Valgrind

Using Lint

Using AddressSanitizer

You neeed clang for this.

"The Debugger"

When I say "the debugger" in the book I mean to use GDB, but use 
every tool you can find that helps.

### Exercise 5 Memorizing C Operators

The Plan

* Learn why memorizing works.
* Learn how to memorize things.
* Review the C operators.

Memorization

* A "backdoor" hack to learning.
* Memorize the operators, then reading is easier.
* Works with any language.

Memorization Proces

* Write everything on index cards.
* Use Anki, but make your own cards.
* Spend 30-60 minutes a day.
* Track what you don't know, drill those more.

Arithmetic Operators

    +    Add
    -    Subtract
    *    Multiply
    /    Divide
    %    Modulus
    ++   Increment
    --   Decrement

Relational Operators

    ==   Equal
    !=   Not equal
    >    Greater than
    <    Less than
    >=   Greater than equal
    <=   Less than equal

Logical Operators

    &&   Logical and
    ||   Logical or
    !    Logical not
    ? :  Logical ternary

Bitwise Operators

    &    Bitwise and
    |    Bitwise or
    ^    Bitwise xor
    ~    Bitwise ones compliment
    <<   Bitwise shift left
    >>   Bitwise shift right

Assignment Operators

    =    Assign equal
    +=   Assign plus-equal
    -=   Assign minus-equal
    *=   Assign multiply-equal
    /=   Assign divide-equal
    %=   Assign modulus-equal
    <<=  Assign shift-left-equal
    >>=  Assign shift-right-equal
    &=   Assign and-equal
    ^=   Assign xor-equal
    |=   Assign or-equal

Data Operators

    sizeof() Get the size of
    []       Array subscript
    &        The address of
    *        The value of
    ->       Structure dereference
    .        Structure reference

Miscellaneous Operators

    ,         Comma
    ( )       Parenthesis
    { }       Braces
    :         Colon
    //        Single-line comment start
    /*        Multi-line comment start
    */        Multi-line comment end

### Exercise 6 Memorizing C Syntax

The Plan

* Memorize the keywords of C.
* Memorize the major syntax forms.

Execution Keywords
    break     Exit out of a compound statement.
    case      A branch in a switch-statement.
    continue  Continue to the top of a loop.
    do        Start a do-while loop.
    default   Default branch in a switch-statement.
    else      An else branch of an if-statement.
    for       Start a for-loop.
    goto      Jump to a label.
    if        Starts an if-statement.
    return    Return from a function.
    switch    Start a switch-statement.
    while     Start a while-loop.

Type Keywords
    char      Character data type.
    double    A double floating point data type.
    float     A floating point data type.
    int       An integer data type.
    long      A long integer data type.
    short     A short integer data type.
    void      Declare a data type empty.
    union     Start a union-statement.
    struct    Combine variables into a single record.

Data Keywords

    auto      Give a local variable a local lifetime.
    const     Make a variable unmodifiable.
    enum      Define a set of int constants.
    extern    Declare an identifier is defined externally.
    register  Declare a variable be stored in a CPU register.
    signed    A signed modifier for integer data types.
    sizeof    Determine the size of data.
    static    Preserve variable value after its scope exits.
    typedef   Create a new type.
    unsigned  An unsigned modifier for integer data types.
    volatile  Declare a variable might be modified elsewhere.

If-Statement

    if(TEST) {
        CODE;
    } else if(TEST) {
        CODE;
    } else {
        CODE;
    }

Switch-Statement

    switch (OPERAND) {
        case CONSTANT:
            CODE;
            break;
        default:
            CODE;
    }

While-Loop

    while(TEST) {
        CODE;
    }

While with Continue

    while(TEST) {
        if(OTHER_TEST) {
            continue;
        }
        CODE;
    }

While with Break

    while(TEST) {
        if(OTHER_TEST) {
            break;
        }
        CODE;
    }

Do-While

    do {
        CODE;
    } while(TEST);

For-Loop

    for(INIT; TEST; POST) {
        CODE;
    }

* *continue* and *break* work with *for*

Enum

    enum { CONST1, CONST2, CONST3 } NAME;

Goto

    if(ERROR_TEST) {
        goto fail;
    }

    fail:
        CODE;

Functions

    TYPE NAME(ARG1, ARG2, ..) {
        CODE;
        return VALUE;
    }

Typedef

    typedef DEFINITION IDENTIFIER;

    typedef unsigned char byte;

Struct

    struct NAME {
        ELEMENTS;
    } [VARIABLE_NAME];

Typedef Struct

    typedef struct [STRUCT_NAME] {
        ELEMENTS;
    } IDENTIFIER;

Union

    union NAME {
        ELEMENTS;
    } [VARIABLE_NAME];

### Exercise 7 Variables and Types

The Plan

* Learn some basic variables and types.
* int, float, double, char, and strings.

The Code

.\ex07\ex7.c

```c
#include <stdio.h>

int main(int argc, char *argv[])
{
    int distance = 100;
    float power = 2.345f;
    double super_power = 56789.4532;
    char initial = 'A';
    char first_name[] = "Zed";
    char last_name[] = "Shaw";

    first_name[3] = 'Z';

    printf("You are %d miles away.\n", distance);
    printf("You have %f levels of power.\n", power);
    printf("You have %f awesome super powers.\n", super_power);
    printf("I have an initial %c.\n", initial);
    printf("I have a first name %s.\n", first_name);
    printf("I have a last name %s.\n", last_name);
    printf("My whole name is %s %c. %s.\n",
            first_name, initial, last_name);

    int bugs = 100;
    double bug_rate = 1.2;

    printf("You have %d bugs at the imaginary rate of %f.\n",
            bugs, bug_rate);

    long universe_of_defects = 1L * 1024L * 1024L * 1024L;
    printf("The entire universe has %ld bugs.\n", universe_of_defects);

    double expected_bugs = bugs * bug_rate;
    printf("You are expected to have %f bugs.\n", expected_bugs);

    double part_of_universe = expected_bugs / universe_of_defects;
    printf("That is only a %e portion of the universe.\n",
            part_of_universe);

    // this makes no sense, just a demo of something weird
    char nul_byte = '\0';
    int care_percentage = bugs * nul_byte;
    printf("Which means you should care %d%%.\n", care_percentage);

    return 0;
}

```

The Analysis

Breaking It

* Strings give us so much more fun now!
* Crafting bad strings.
* Messing with pointers.
* Abusing printf.

Extra Credit

* Make the number you assign to *universe_of_defects* various
  sizes until you get a warning from the compiler.
* What do these really huge numbers actually print out?
* Change *long* to *unsigned long* and try to find
  the number that makes it too big.
* Go search online to find out what *unsigned* does.
* Try to explain to yourself (before I do in the next exercise)
  why you can multiply a *char* and an *int*.

### Exercise 8 If, Else-If, Else

The Plan

Simply learn to use this:

    if(TEST) {
        CODE;
    } else if(TEST) {
        CODE;
    } else {
        CODE;
    }

The Code

.\ex08\ex8.c

```c
#include <stdio.h>

int main(int argc, char *argv[])
{
    int i = 0;

    if (argc == 1) {
        printf("You only have one argument. You suck.\n");
    } else if (argc > 1 && argc < 4) {
        printf("Here's your arguments:\n");

        for (i = 0; i < argc; i++) {
            printf("%s ", argv[i]);
        }
        printf("\n");
    } else if (argc > 10) {
        printf("You have too many arguments. You suck.\n");
    }

    return 0;
}

```

The Analysis

Breaking It

* It kind of just works, but remove the *else* and change the logic.

Extra Credit

* You were briefly introduced to *&&*, which does an *and* comparison,
  so go research online the different *Boolean operators*.
* Write a few more test cases for this program to see what you can come
  up with.

### Exercise 9 While-Loop and Boolean Expressions

The Plan

You first loop shall be the *while*:

    while(TEST) {
        CODE;
    }

The Code

.\ex09\ex9.c

```c
#include <stdio.h>

int main(int argc, char *argv[])
{
    int i;
    while (i < 25) {
        printf("%d\n", i);
        i++;
    }

    return 0;
}

```

The Analysis

Breaking It

* Forget to initialize the *int i*.
* Forget to do an i++ and make it run forever.

Extra Credit

* Make the loop count backward by using ``i--`` to start
  at 25 and go to 0.
* Write a few more complex ``while-loops`` using what you know
  so far.

### Exercise 10 Switch Statements

The Plan

* Learn about the *switch-statement* and indirectly jump tables.
* Write a program that takes a command line argument.

The Code

.\ex10\ex10.c

```c
#include <stdio.h>

int main(int argc, char *argv[])
{
    if (argc != 2) {
        printf("ERROR: You need one argument.\n");
        // this is how you abort a program
        return 1;
    }

    int i = 0;
    for (i = 0; argv[1][i] != '\0'; i++) {
        char letter = argv[1][i];

        switch (letter) {
            case 'a':
            case 'A':
                printf("%d: 'A'\n", i);
                break;

            case 'e':
            case 'E':
                printf("%d: 'E'\n", i);
                break;

            case 'i':
            case 'I':
                printf("%d: 'I'\n", i);
                break;

            case 'o':
            case 'O':
                printf("%d: 'O'\n", i);
                break;

            case 'u':
            case 'U':
                printf("%d: 'U'\n", i);
                break;

            case 'y':
            case 'Y':
                if (i > 2) {
                    // it's only sometimes Y
                    printf("%d: 'Y'\n", i);
                }
                break;

            default:
                printf("%d: %c is not a vowel\n", i, letter);
        }
    }

    return 0;
}

```

The Analysis

Let's talk about jump tables, in the naive sense.

Breaking It

* Forget a *break*, and it'll run two or more blocks of code you don't want it to run.
* Forget a *default*, and it'll silently ignore values you forgot.
* Accidentally put a variable into the *switch* that evaluates to something unexpected, like an *int*, which becomes weird values.
* Use uninitialized values in the *switch*.

Extra Credit

* Write another program that uses math on the letter to
  convert it to lowercase, and then remove all of the extraneous
  uppercase letters in the switch.
* Use the *','* (comma) to initialize *letter*
  in the *for-loop*.
* Make it handle all of the arguments you pass it with
  yet another *for-loop*.

Extra Credit

* Convert this *switch-statement* to an *if-statement*.
  Which do you like better?
* In the case for 'Y' I have the break outside of the *if-statement*. What's the impact of this,
  and what happens if you move it inside of the *if-statement*. Prove to yourself that you're right.

### Exercise 11 Arrays and Strings

The Plan

* Learn the similarity between arrays and strings.
* Avoid getting pedantic about them.
* Learn how C stores strings and processes them.

The Code

.\ex11\ex11.c

```c
#include <stdio.h>

int main(int argc, char *argv[])
{
    int numbers[4] = { 0 };
    char name[4] = { 'a', 'a', 'a', 'a' };

    // first, print them out raw
    printf("numbers: %d %d %d %d\n",
            numbers[0], numbers[1], numbers[2], numbers[3]);

    printf("name each: %c %c %c %c\n",
            name[0], name[1], name[2], name[3]);

    printf("name: %s\n", name);

    // setup the numbers
    numbers[0] = 1;
    numbers[1] = 2;
    numbers[2] = 3;
    numbers[3] = 4;

    // setup the name
    name[0] = 'Z';
    name[1] = 'e';
    name[2] = 'd';
    name[3] = 'A';

    // then print them out initialized
    printf("numbers: %d %d %d %d\n",
            numbers[0], numbers[1], numbers[2], numbers[3]);

    printf("name each: %c %c %c %c\n",
            name[0], name[1], name[2], name[3]);

    // print the name like a string
    printf("name: %s\n", name);

    // another way to use name
    char *another = "Zed";

    printf("another: %s\n", another);

    printf("another each: %c %c %c %c\n",
            another[0], another[1], another[2], another[3]);

    return 0;
}

```

The Analysis

Breaking It

So many ways to break this!

* Get rid of the initializers that set up *name*.
* Accidentally set *name[3] = 'A';* so that there's no terminator.
* Set the initializer to *{'a','a','a','a'}* so that there are too many
  'a' characters and no space for the *'\0'* terminator.

Extra Credit

* Assign the characters into *numbers*, and then use *printf*
  to print them one character at a time.  What kind of compiler warnings
  do you get?
* Do the inverse for *name*, trying to treat it like an array
  of *int* and print it out one *int* at a time.  What
  does the debugger think of that?
* In how many other ways can you print this out?

Extra Credit

* If an array of characters is 4 bytes long, and an integer is 4 bytes
  long, then can you treat the whole *name* array like it's just
  an integer?  How might you accomplish this crazy hack?
* Take out a piece of paper and draw each of these arrays as a
  row of boxes. Then do the operations you just did on paper to see
  if you get them right.
* Convert *name* to be in the style of *another* and see
  if the code keeps working.

### Exercise 12 Sizes and Arrays

The Plan

* Learn about *sizeof* and how it relates to arrays.

The Code

.\ex12\ex12.c

```c
#include <stdio.h>

int main(int argc, char *argv[])
{
    int areas[] = { 10, 12, 13, 14, 20 };
    char name[] = "Zed";
    char full_name[] = {
        'Z', 'e', 'd',
        ' ', 'A', '.', ' ',
        'S', 'h', 'a', 'w' 
    };

    // WARNING: On some systems you may have to change the
    // %ld in this code to a %u since it will use unsigned ints
    printf("The size of an int: %ld\n", sizeof(int));
    printf("The size of areas (int[]): %ld\n", sizeof(areas));
    printf("The number of ints in areas: %ld\n",
            sizeof(areas) / sizeof(int));
    printf("The first area is %d, the 2nd %d.\n", areas[0], areas[1]);

    printf("The size of a char: %ld\n", sizeof(char));
    printf("The size of name (char[]): %ld\n", sizeof(name));
    printf("The number of chars: %ld\n", sizeof(name) / sizeof(char));

    printf("The size of full_name (char[]): %ld\n", sizeof(full_name));
    printf("The number of chars: %ld\n",
            sizeof(full_name) / sizeof(char));

    full_name[12] = 'X';

    printf("name=\"%s\" and full_name=\"%s\"\n", name, full_name);

    return 0;
}

```

The Analysis

Breaking It

* Get rid of the *'\0'* at the end of *full_name*
  and re-run it.  Run it under the debugger, too.  Now, move the definition
  of *full_name* to the top of *main* before *areas*.
  Try running it under the debugger a few times and see if you get some
  new errors.  In some cases, you might still get lucky and not catch
  any errors.
* Change it so that instead of *areas[0]* you try to
  print *areas[10]*.  See what the debugger thinks of that.
* Try other ways to break it like this, doing it to *name* and
  *full_name* too.

Extra Credit

* Try assigning to elements in the *areas* array with *areas[0] = 100;* and similar.
* Try assigning to elements of *name* and *full_name*.
* Try setting one element of *areas* to a character from *name*.
* Search online for the different sizes used for integers on different
  CPUs.

### Exercise 13 For-Loops and Arrays of Strings

The Plan

Learn about this code:

    for(INITIALIZER; TEST; INCREMENTER) {
        CODE;
    }

The Code

.\ex13\ex13.c

```c
#include <stdio.h>

int main(int argc, char *argv[])
{
    int i = 0;

    // go through each string in argv
    // why am I skipping argv[0]?
    for (i = 0; i < argc; i++) {
        printf("arg %d: %s\n", i, argv[i]);
    }

    // let's make our own array of strings
    char *states[] = {
        "California", "Oregon",
        "Washington", "Texas"
    };

    int num_states = 5;

    for (i = 0; i < num_states; i++) {
        printf("state %d: %s\n", i, states[i]);
    }

    return 0;
}

```

The Analysis

Breaking It

* Take your favorite other language and use it to run this program, but include as many command line arguments as possible.  See if you can bust it
  by giving it way too many arguments.
* Initialize *i* to 0 and see what that does.  Do you have to adjust
  *argc* as well, or does it just work?  Why does 0-based indexing work
  here?
* Set *num_states* wrong so that it's a higher value and see what
  it does.

Extra Credit

* Figure out what kind of code you can put into the parts of a *for-loop*.
* Look up how to use the comma character (,) to separate multiple
  statements in the parts of the *for-loop*, but between the semicolon characters (;).
* Read what a *NULL* is and try to use it in one of the elements from the
  *states* array to see what it'll print.
* See if you can assign an element from the *states* array to the
  *argv* array before printing both.  Try the inverse.

### Exercise 14 Writing and Using Functions

The Plan

* Write your very first functions.

The Code

.\ex14\ex14.c

```c
#include <stdio.h>
#include <ctype.h>

// forward declarations
int can_print_it(char ch);
void print_letters(char arg[]);

void print_arguments(int argc, char *argv[])
{
    int i = 0;

    for (i = 0; i < argc; i++) {
        print_letters(argv[i]);
    }
}

void print_letters(char arg[])
{
    int i = 0;

    for (i = 0; arg[i] != '\0'; i++) {
        char ch = arg[i];

        if (can_print_it(ch)) {
            printf("'%c' == %d ", ch, ch);
        }
    }

    printf("\n");
}

int can_print_it(char ch)
{
    return isalpha(ch) || isblank(ch);
}

int main(int argc, char *argv[])
{
    print_arguments(argc+1, argv);
    return 0;
}

```

The Analysis

Breaking It

* Remove the forward declarations to confuse the compiler and cause it to complains about *can_print_it* and *print_letters*.
* When you call *print_arguments* inside *main*, try
  adding 1 to *argc* so that it goes past the end of the
  *argv* array.

Extra Credit

* Rework these functions so that you have fewer functions.  For example,
  do you really need *can_print_it*?
* Have *print_arguments* figure out how long each argument string
  is by using the *strlen* function, and then pass that length
  to *print_letters*.  Then, rewrite *print_letters*
  so it only processes this fixed length and doesn't rely on the
  *'\0'* terminator.  You'll need the *#include <string.h>* for this.

Extra Credit

* Use *man* to look up information on *isalpha*
  and *isblank*.  Use other similar functions to
  print out only digits or other characters.
* Go read about how other people like to format their
  functions.  Never use the *K&R syntax* (it's antiquated and
  confusing) but understand what it's doing in case you run
  into someone who likes it.

### Exercise 15 Pointers, Dreaded Pointers

The Plan

* A long video on C pointers.
* Lots of demonstration and visuals.

The Code

.\ex15\ex15.c

```c
#include <stdio.h>

int main(int argc, char *argv[])
{
    // create two arrays we care about
    int ages[] = { 23, 43, 12, 89, 2 };
    char *names[] = {
        "Alan", "Frank",
        "Mary", "John", "Lisa"
    };

    // safely get the size of ages
    int count = sizeof(ages) / sizeof(int);
    int i = 0;

    // first way using indexing
    for (i = 0; i < count; i++) {
        printf("%s has %d years alive.\n", names[i], ages[i]);
    }

    printf("---\n");

    // setup the pointers to the start of the arrays
    int *cur_age = (int *)names;
    char **cur_name = names;

    // second way using pointers
    for (i = 0; i < count; i++) {
        printf("%s is %d years old.\n",
                *(cur_name + i), *(cur_age + i));
    }

    printf("---\n");

    // third way, pointers are just arrays
    for (i = 0; i < count; i++) {
        printf("%s is %d years old again.\n", cur_name[i], cur_age[i]);
    }

    printf("---\n");

    // fourth way with pointers in a stupid complex way
    for (cur_name = names, cur_age = ages;
            (cur_age - ages) < count; cur_name++, cur_age++) {
        printf("%s lived %d years so far.\n", *cur_name, *cur_age);
    }

    return 0;
}

```

The Pointer Lexicon

    type *ptr A pointer of type named ptr
    *ptr The value of whatever ptr is pointed at
    *(ptr + i) The value of (whatever ptr is pointed at plus i)
    &thing The address of thing
    type *ptr = &thing A pointer of type named ptr set to the address of thing
    ptr++ Increment where ptr points

Pointers Visually

The Analysis

Breaking It

* Try to make *cur_age* point at *names*.  You'll need to
  use a C cast to force it, so go look that up and try to figure it out.
* In the final *for-loop*, try getting the math wrong in weird ways.
* Try rewriting the loops so that they start at the end of the arrays and go
  to the beginning.  This is harder than it looks.

Extra Credit

* Rewrite all of the arrays in this program as pointers.
* Rewrite all of the pointers as arrays.
* Go back to some of the other programs that use arrays and
  try to use pointers instead.
* Process command line arguments using just pointers similar to how
  you did *names* in this one.
* Play with combinations of getting the value of and the address of
  things.
* Add another *for-loop* at the end that prints out the
  addresses that these pointers are using.  You'll need the *%p* format
  for *printf*.

Extra Credit

* Rewrite this program to use a function for each of the ways you're
  printing out things.  Try to pass pointers to these functions so that
  they work on the data.  Remember you can declare a function to accept
  a pointer, but just use it like an array.
* Change the *for-loops* to *while-loops* and see what
  works better for which kind of pointer usage.

### Exercise 16 Structs And Pointers To Them

The Plan

* Learn to work with *structs* to structure data and make new types.
* Learn to use pointers to work with *structs* better.

The Code

.\ex16\ex16.c

```c
#include <stdio.h>
#include <assert.h>
#include <stdlib.h>
#include <string.h>

struct Person {
    char *name;
    int age;
    int height;
    int weight;
};

struct Person *Person_create(char *name, int age, int height,
        int weight)
{
    struct Person *who = malloc(sizeof(struct Person));
    assert(who != NULL);

    who->name = strdup(name);
    who->age = age;
    who->height = height;
    who->weight = weight;

    return who;
}

void Person_destroy(struct Person *who)
{
    assert(who != NULL);

    free(who->name);
    free(who);
}

void Person_print(struct Person *who)
{
    printf("Name: %s\n", who->name);
    printf("\tAge: %d\n", who->age);
    printf("\tHeight: %d\n", who->height);
    printf("\tWeight: %d\n", who->weight);
}

int main(int argc, char *argv[])
{
    // make two people structures
    struct Person *joe = Person_create("<NAME>", 32, 64, 140);

    struct Person *frank = Person_create("<NAME>", 20, 72, 180);

    // print them out and where they are in memory
    printf("Joe is at memory location %p:\n", joe);
    Person_print(joe);

    printf("Frank is at memory location %p:\n", frank);
    Person_print(frank);

    // make everyone age 20 years and print them again
    joe->age += 20;
    joe->height -= 2;
    joe->weight += 40;
    Person_print(joe);

    frank->age += 20;
    frank->weight += 20;
    free(frank);
    Person_print(frank);

    // destroy them both so we clean up
    Person_destroy(joe);
    Person_destroy(frank);

    return 0;
}

```

The Analysis

Breaking It

* Try passing *NULL* to *Person_destroy* see what
  it does.  If it doesn't abort, then you must not have the
  *-g* option in your Makefile's *CFLAGS*.
* Forget to call *Person_destroy* at the end, and then run
  it under the debugger to see it report that you forgot
  to free the memory.  Figure out the options you need to pass
  to the debugger to get it to print how you leaked
  this memory.

Breaking It

* Forget to free *who->name* in *Person_destroy*
  and compare the output.  Again, use the right options to
  see how the debugger tells you exactly where you messed
  up.
* This time, pass *NULL* to *Person_print* and
  see what the debugger thinks of that. You'll figure out that *NULL* is a quick way
  to crash your program.

Extra Credit

* How to create a *struct* on the *stack* just like you're making any other variable.
* How to initialize it using the *x.y* (period) character
  instead of the *x->y* syntax.
* How to pass a structure to other functions without using
  a pointer.

### Exercise 17 Heap and Stack Memory Allocation

The Plan

* Learn to allocate data on the heap using *malloc*.
* Memory management techniques to avoid leaking.
* How the heap differs from the stack, and when to use them.

The Code

.\ex17\ex17.c

```c
#include <stdio.h>
#include <assert.h>
#include <stdlib.h>
#include <errno.h>
#include <string.h>

#define MAX_DATA 512
#define MAX_ROWS 100

struct Address {
    int id;
    int set;
    char name[MAX_DATA];
    char email[MAX_DATA];
};

struct Database {
    struct Address rows[MAX_ROWS];
};

struct Connection {
    FILE *file;
    struct Database *db;
};

void die(const char *message)
{
    if (errno) {
        perror(message);
    } else {
        printf("ERROR: %s\n", message);
    }

    exit(1);
}

void Address_print(struct Address *addr)
{
    printf("%d %s %s\n", addr->id, addr->name, addr->email);
}

void Database_load(struct Connection *conn)
{
    int rc = fread(conn->db, sizeof(struct Database), 1, conn->file);
    if (rc != 1)
        die("Failed to load database.");
}

struct Connection *Database_open(const char *filename, char mode)
{
    struct Connection *conn = malloc(sizeof(struct Connection));
    if (!conn)
        die("Memory error");

    conn->db = malloc(sizeof(struct Database));
    if (!conn->db)
        die("Memory error");

    if (mode == 'c') {
        conn->file = fopen(filename, "w");
    } else {
        conn->file = fopen(filename, "r+");

        if (conn->file) {
            Database_load(conn);
        }
    }

    if (!conn->file)
        die("Failed to open the file");

    return conn;
}

void Database_close(struct Connection *conn)
{
    if (conn) {
        if (conn->file)
            fclose(conn->file);
        if (conn->db)
            free(conn->db);
        free(conn);
    }
}

void Database_write(struct Connection *conn)
{
    rewind(conn->file);

    int rc = fwrite(conn->db, sizeof(struct Database), 1, conn->file);
    if (rc != 1)
        die("Failed to write database.");

    rc = fflush(conn->file);
    if (rc == -1)
        die("Cannot flush database.");
}

void Database_create(struct Connection *conn)
{
    int i = 0;

    for (i = 0; i < MAX_ROWS; i++) {
        // make a prototype to initialize it
        struct Address addr = {.id = i,.set = 0 };
        // then just assign it
        conn->db->rows[i] = addr;
    }
}

void Database_set(struct Connection *conn, int id, const char *name,
        const char *email)
{
    struct Address *addr = &conn->db->rows[id];
    if (addr->set)
        die("Already set, delete it first");

    addr->set = 1;
    // WARNING: bug, read the "How To Break It" and fix this
    char *res = strncpy(addr->name, name, MAX_DATA);
    // demonstrate the strncpy bug
    if (!res)
        die("Name copy failed");

    res = strncpy(addr->email, email, MAX_DATA);
    if (!res)
        die("Email copy failed");
}

void Database_get(struct Connection *conn, int id)
{
    struct Address *addr = &conn->db->rows[id];

    if (addr->set) {
        Address_print(addr);
    } else {
        die("ID is not set");
    }
}

void Database_delete(struct Connection *conn, int id)
{
    struct Address addr = {.id = id,.set = 0 };
    conn->db->rows[id] = addr;
}

void Database_list(struct Connection *conn)
{
    int i = 0;
    struct Database *db = conn->db;

    for (i = 0; i < MAX_ROWS; i++) {
        struct Address *cur = &db->rows[i];

        if (cur->set) {
            Address_print(cur);
        }
    }
}

int main(int argc, char *argv[])
{
    if (argc < 3)
        die("USAGE: ex17 <dbfile> <action> [action params]");

    char *filename = argv[1];
    char action = argv[2][0];
    struct Connection *conn = Database_open(filename, action);
    int id = 0;

    if (argc > 3) id = atoi(argv[3]);
    if (id >= MAX_ROWS) die("There's not that many records.");

    switch (action) {
        case 'c':
            Database_create(conn);
            Database_write(conn);
            break;

        case 'g':
            if (argc != 4)
                die("Need an id to get");

            Database_get(conn, id);
            break;

        case 's':
            if (argc != 6)
                die("Need id, name, email to set");

            Database_set(conn, id, argv[4], argv[5]);
            Database_write(conn);
            break;

        case 'd':
            if (argc != 4)
                die("Need id to delete");

            Database_delete(conn, id);
            Database_write(conn);
            break;

        case 'l':
            Database_list(conn);
            break;
        default:
            die("Invalid action: c=create, g=get, s=set, d=del, l=list");
    }

    Database_close(conn);

    return 0;
}

```

```bash
$ make ex1. cc -Wall ­g ex17.c -o ex1. 
$ ./ex17 db.dat c 
$ ./ex17 db.dat s 1 zed <EMAIL> 
$ ./ex17 db.dat s 2 frank <EMAIL>$ 
$ ./ex17 db.dat s 3 joe <EMAIL> 
$ 
$ ./ex17 db.dat l 
1 zed <EMAIL> 
2 frank <EMAIL> 
3 joe <EMAIL> 

$ ./ex17 db.dat d 3 
$ ./ex17 db.dat l 
1 zed <EMAIL> 
2 frank <EMAIL> 

$ ./ex17 db.dat g 2 
2 frank <EMAIL> 
```

The Analysis

Breaking It

* The classic way is to remove some of the safety checks so that you can
  pass in arbitrary data. For example, remove the check on line 160
  that prevents you from passing in any record number.
* You can also try corrupting the data file.  Open it in any editor and
  change random bytes, and then close it.
* You could also find ways to pass bad arguments to the program when it's
  run. For example, getting the file and action backwards will make it create
  a file named after the action, and then do an action based on the first
  character.

Breaking It

* There's a bug in this program because *strncpy* is poorly
  designed.  Go read about *strncpy* and try to find out what happens
  when the *name* or *address* you give is *greater* than
  512 bytes.  Fix this by simply forcing the last character to *'\0'*
  so that it's always set no matter what (which is what strncpy should do).
* In the extra credit, I have you augment the program to create arbitrary
  size databases.  Try to see what the biggest database is before you
  cause the program to die due to lack of memory from *malloc*.

Extra Credit

* The *die* function needs to be augmented to let you pass the *conn*
  variable, so it can close it and clean up.
* Change the code to accept parameters for *MAX_DATA* and *MAX_ROWS*, store them in the *Database* struct, and write that to the file, thus creating
  a database that can be arbitrarily sized.
* Add more operations you can do with the database, like *find*.

Extra Credit

* Read about how C does it's struct packing, and then try to see why your
  file is the size it is.  See if you can calculate a new size after adding
  more fields.
* Add some more fields to *Address* and make them searchable.
* Write a shell script that will do your testing automatically for you
  by running commands in the right order.  Hint: Use *set -e* at the
  top of a *bash* to make it abort the whole script if any command
  has an error.

Extra Credit

* Try reworking the program to use a single global for the database connection.
  How does this new version of the program compare to the other one?
* Go research stack data structure and write one in your favorite language,
  then try to do it in C.

### Exercise 18 Pointers to Functions

The Plan

* Advanced topic of pointers to functions.
* These are very useful but not encountered too often.

The Code

.\ex18\ex18.c

```c
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <string.h>

/** Our old friend die from ex17. */
void die(const char *message)
{
    if (errno) {
        perror(message);
    } else {
        printf("ERROR: %s\n", message);
    }

    exit(1);
}

// a typedef creates a fake type, in this
// case for a function pointer
typedef int (*compare_cb) (int a, int b);

/**
 * A classic bubble sort function that uses the 
 * compare_cb to do the sorting. 
 */
int *bubble_sort(int *numbers, int count, compare_cb cmp)
{
    int temp = 0;
    int i = 0;
    int j = 0;
    int *target = malloc(count * sizeof(int));

    if (!target)
        die("Memory error.");

    memcpy(target, numbers, count * sizeof(int));

    for (i = 0; i < count; i++) {
        for (j = 0; j < count - 1; j++) {
            if (cmp(target[j], target[j + 1]) > 0) {
                temp = target[j + 1];
                target[j + 1] = target[j];
                target[j] = temp;
            }
        }
    }

    return target;
}

int sorted_order(int a, int b)
{
    return a - b;
}

int reverse_order(int a, int b)
{
    return b - a;
}

int strange_order(int a, int b)
{
    if (a == 0 || b == 0) {
        return 0;
    } else {
        return a % b;
    }
}

/** 
 * Used to test that we are sorting things correctly
 * by doing the sort and printing it out.
 */
void test_sorting(int *numbers, int count, compare_cb cmp)
{
    int i = 0;
    int *sorted = bubble_sort(numbers, count, cmp);

    if (!sorted)
        die("Failed to sort as requested.");

    for (i = 0; i < count; i++) {
        printf("%d ", sorted[i]);
    }
    printf("\n");

    free(sorted);
}

void destroy(compare_cb cmp)
{
    int i = 0;

    unsigned char *data = (unsigned char *)cmp;

    for(i = 0; i < 1; i++) {
        data[i] = i;
    }

    printf("\n");
}

void dump(compare_cb cmp)
{
    int i = 0;

    unsigned char *data = (unsigned char *)cmp;

    for(i = 0; i < 25; i++) {
        printf("%02x:", data[i]);
    }

    printf("\n");
}

int main(int argc, char *argv[])
{
    if (argc < 2) die("USAGE: ex18 4 3 1 5 6");

    int count = argc - 1;
    int i = 0;
    char **inputs = argv + 1;

    int *numbers = malloc(count * sizeof(int));
    if (!numbers) die("Memory error.");

    for (i = 0; i < count; i++) {
        numbers[i] = atoi(inputs[i]);
    }

    test_sorting(numbers, count, sorted_order);
    test_sorting(numbers, count, reverse_order);
    test_sorting(numbers, count, strange_order);

    free(numbers);

    printf("SORTED:");
    dump(sorted_order);

    destroy(sorted_order);

    printf("SORTED:");
    dump(sorted_order);

    return 0;
}

```

The Analysis

Breaking It

Let's hack your computer with this code:

    unsigned char *data = (unsigned char *)cmp;

    for(i = 0; i < 25; i++) {
        printf("%02x:", data[i]);
    }

    printf("\n");

You'll see how the bytes of code that make up your program can also be data.

Extra Credit

* Get a hex editor and open up *ex18*, and then find the sequence
  of hex digits that start a function to see if you can find the function
  in the raw program.
* Find other random things in your hex editor and change them.  Rerun your
  program and see what happens.  Strings you find are the easiest
  things to change.
* Pass in the wrong function for the *compare_cb* and see what
  the C compiler complains about.
* Pass in NULL and watch your program seriously bite it.  Then, run
  the debugger and see what that reports.
* Write another sorting algorithm, then change *test_sorting* so
  that it takes *both* an arbitrary sort function and the sort function's
  callback comparison.  Use it to test both of your algorithms.

### Exercise 19 Zed's Awesome Debug Macros

The Plan

* Learn about the macros that vastly improve my code quality.
* Find out why they help you out.
* Explore some advanced C Pre-Processor (CPP) macro magic code generation tricks.

The Code

.\ex19\ex19.c

```c

#include "dbg.h"
#include <stdlib.h>
#include <stdio.h>

void test_debug()
{
    // notice you don't need the \n
    debug("I have Brown Hair.");

    // passing in arguments like printf
    debug("I am %d years old.", 37);
}

void test_log_err()
{
    log_err("I believe everything is broken.");
    log_err("There are %d problems in %s.", 0, "space");
}

void test_log_warn()
{
    log_warn("You can safely ignore this.");
    log_warn("Maybe consider looking at: %s.", "/etc/passwd");
}

void test_log_info()
{
    log_info("Well I did something mundane.");
    log_info("It happened %f times today.", 1.3f);
}

int test_check(char *file_name)
{
    FILE *input = NULL;
    char *block = NULL;

    block = malloc(100);
    check_mem(block);        // should work

    input = fopen(file_name, "r");
    check(input, "Failed to open %s.", file_name);

    free(block);
    fclose(input);
    return 0;

error:
    if (block) free(block);
    if (input) fclose(input);
    return -1;
}

int test_sentinel(int code)
{
    char *temp = malloc(100);
    check_mem(temp);

    switch (code) {
        case 1:
            log_info("It worked.");
            break;
        default:
            sentinel("I shouldn't run.");
    }

    free(temp);
    return 0;

error:
    if (temp)
        free(temp);
    return -1;
}

int test_check_mem()
{
    char *test = NULL;
    check_mem(test);

    free(test);
    return 1;

error:
    return -1;
}

int test_check_debug()
{
    int i = 0;
    check_debug(i != 0, "Oops, I was 0.");

    return 0;
error:
    return -1;
}

int main(int argc, char *argv[])
{
    check(argc == 2, "Need an argument.");

    test_debug();
    test_log_err();
    test_log_warn();
    test_log_info();

    check(test_check("ex20.c") == 0, "failed with ex20.c");
    check(test_check(argv[1]) == -1, "failed with argv");
    check(test_sentinel(1) == 0, "test_sentinel failed.");
    check(test_sentinel(100) == -1, "test_sentinel failed.");
    check(test_check_mem() == -1, "test_check_mem failed.");
    check(test_check_debug() == -1, "test_check_debug failed.");

    return 0;

error:
    return 1;
}

```

The Analysis

Breaking It

These macros are designed on purpose to prevent you from doing this:

    if(blah) debug("This is a thing");
    else debug ("This is another thing");

Extra Credit

* Put ``#define NDEBUG`` at the top of the file and check that all
  of the debug messages go away.
* Undo that line, and add ``-DNDEBUG`` to ``CFLAGS`` at the
  top of the ``Makefile``, and then recompile to see the same thing.
* Modify the logging so that it includes the function name, as well
  as the ``file:line``.

### Exercise 20 Advanced Debugging Techniques

The Plan

Demonstrate more advanced debugging techniques and tools.

The Demonstration

Extra Credit

* Find a graphical debugger and compare using it to raw ``gdb``.
  These are useful when the program you're looking at is local, but they
  are pointless if you have to debug a program on a server.
* You can enable core dumps on your OS, and when a program crashes,
  you'll get a core file.  This core file is like a postmortem of
  the program that you can load up to see what happened right at the crash
  and what caused it.  Change ``ex31.c`` so that it crashes
  after a few iterations, then try to get a core dump and analyze it.

### Exercise 21 Advanced Data Types and Flow Control

The Plan

* Learn about the basic types and keywords for them.
* Cover all the keywords for modifying those types.
* Review fixed exact size types.
* Learn all the different operators on those types.

This is mostly a review!

Available Data Types

    int    Stores a regular integer, defaulting to 32 bits in size.
    double Holds a large floating point number.
    float  Holds a smaller floating point number.
    char   Holds a single 1 byte character.
    void   Indicates "no type".
    enum   Enumerated types, which work as and convert to integers.

Type Modifiers

    unsigned  Non-negative numbers.
    signed    Gives you negative and positive numbers.
    long      Bigger number.
    short     Smaller number.

Type Qualifiers

    const     Constant.
    volatile  Compiler can't trust it.
    register  Put it in a CPU register.

Type Conversion

C type promotion order:

* long double
* double
* float
* int (but only char and short int);
* long

When in doubt, parens it out!

Exact Size Types

If you need exact sizes use these:

    int8_t   8-bit signed integer
    uint8_t  8-bit unsigned integer
    int16_t  16-bit signed integer
    uint16_t 16-bit unsigned integer
    int32_t  32-bit signed integer
    uint32_t 32-bit unsigned integer
    int64_t  64-bit signed integer
    uint64_t 64-bit unsigned integer

Getting Sizes

Refer to the book as there's a large number of
macros to help you get size information for types.

Examples:

    int_least32_t  int that holds at least 32 bits.
    uint_fast32_t  unsigned fastest int for 32 bits.
    intptr_t       signed int that can hold a pointer.
    PTRDIFF_MAX    maximum value of ptrdiff_t
    SIZE_MAX       maximum value of a size_t

Available Operators

This section is a review of what you memorized already
to make sure you know everything.

Memorize these again to be sure you have them.

Extra Credit

* Read stdint.h or a description of it, and write out all the
  available size identifiers.
* Go through each item here and write out what it does in code.  Research it online so you know you got it right.
* Get this information memorized by making flash cards and spending 15
  minutes a day practicing it.
* Create a program that prints out examples of each type, and confirm that your
  research is right.

### Exercise 22 The Stack, Scope, and Globals

The Plan

* Start to learn about scope.
* Stack vs. global.
* Scope levels inside a function.
* The *extern* keyword.

The Code

.\ex22\ex22.h

```c
#ifndef _ex22_h
#define _ex22_h

struct State {
    int the_size;
    int the_age;
};

// gets and sets an internal static variable in ex22.c
int get_age(struct State *state);
void set_age(struct State *state, int age);

// updates a static variable that's inside update_ratio
double update_ratio(double ratio);

void print_size();

#endif

```

.\ex22\ex22.c

```c
#include <stdio.h>
#include "ex22.h"
#include "dbg.h"

int get_age(struct State *state)
{
    return state->the_age;
}

void set_age(struct State *state, int age)
{
   state->the_age = age; 
}

double update_ratio(double new_ratio)
{
    static double ratio = 1.0;

    double old_ratio = ratio;
    ratio = new_ratio;

    return old_ratio;
}

void print_size()
{
    log_info("I think size is: %d", THE_SIZE);
}

```

.\ex22\ex22_main.c

```c
#include "ex22.h"
#include "dbg.h"

const char *MY_NAME = "<NAME>";

void scope_demo(int count)
{
    log_info("count is: %d", count);

    if (count > 10) {
        int numbers = 100;    // BAD! BUGS!

        log_info("count in this scope is %d", numbers);
    }

    log_info("count is at exit: %d", count);

    count = 3000;

    log_info("count after assign: %d", count);
}

int main(int argc, char *argv[])
{
    // test out THE_AGE accessors
    log_info("My name: %s, age: %d", MY_NAME, get_age());

    set_age(100);

    log_info("My age is now: %d", get_age());

    // test out THE_SIZE extern
    log_info("THE_SIZE is: %d", THE_SIZE);
    print_size();

    THE_SIZE = 9;

    log_info("THE SIZE is now: %d", THE_SIZE);
    print_size();

    // test the ratio function static
    log_info("Ratio at first: %f", update_ratio(2.0));
    log_info("Ratio again: %f", update_ratio(10.0));
    log_info("Ratio once more: %f", update_ratio(300.0));

    // test the scope demo
    int count = 4;
    scope_demo(count);
    scope_demo(count * 20);

    log_info("count after calling scope_demo: %d", count);

    return 0;
}

```

This exercises requires two files:

    * ex22.c
    * ex22_main.c

The Analysis

Fixing It

Instead of breaking this one I'm going to fix it.

* Do not shadow a variable like *count* on ex22_main.c:11.
* Avoid using too many globals.
* When in doubt, put it on the heap (malloc).
* Don't use function static variables like I did in ex22.c:update_ratio.
* Avoid reusing function parameters.

Breaking It

* Try to directly access variables in ``ex22.c`` from ``ex22_main.c``
  that you think you can't.  For example, can you get at ``ratio``
  inside ``update_ratio``? What if you had a pointer to it?
* Ditch the ``extern`` declaration in ``ex22.h`` to see what
  errors or warnings you get.
* Add ``static`` or ``const`` specifiers to different variables,
  and then try to change them.

Extra Credit

* Research the concept of pass by value verses pass by reference.  Write an
  example of both.
* Use pointers to gain access to things you shouldn't have access to.
* Use your debugger to see what this kind of access looks like when you
  do it wrong.
* Write a recursive function that causes a stack overflow.  Don't know
  what a recursive function is?  Try calling ``scope_demo`` at the
  bottom of ``scope_demo`` itself so that it loops.
* Rewrite the ``Makefile`` so that it can build this.

### Exercise 23 Meet Duff's Device

The Plan

Learn the most evil awesome hack ever:

Duff's Device

The Code

.\ex23\ex23.c

```c
#include <stdio.h>
#include <string.h>
#include "dbg.h"

int normal_copy(char *from, char *to, int count)
{
    int i = 0;

    for (i = 0; i < count; i++) {
        to[i] = from[i];
    }

    return i;
}

int duffs_device(char *from, char *to, int count)
{
    {
        int n = (count + 7) / 8;

        switch (count % 8) {
            case 0:
                do {
                    *to++ = *from++;
                    case 7:
                    *to++ = *from++;
                    case 6:
                    *to++ = *from++;
                    case 5:
                    *to++ = *from++;
                    case 4:
                    *to++ = *from++;
                    case 3:
                    *to++ = *from++;
                    case 2:
                    *to++ = *from++;
                    case 1:
                    *to++ = *from++;
                } while (--n > 0);
        }
    }

    return count;
}

int zeds_device(char *from, char *to, int count)
{
    {
        int n = (count + 7) / 8;
        debug("n starts: %d, count: %d, count%%8: %d", 
                n, count, count % 8);

        switch (count % 8) {
            case 0:
again:    *to++ = *from++;

            case 7:
          *to++ = *from++;
            case 6:
          *to++ = *from++;
            case 5:
          *to++ = *from++;
            case 4:
          *to++ = *from++;
            case 3:
          *to++ = *from++;
            case 2:
          *to++ = *from++;
            case 1:
          *to++ = *from++;
          debug("last case: n=%d", n);
          if (--n > 0) {
              debug("going again: n=%d", n);
              goto again;
          }
        }
    }

    return count;
}

int valid_copy(char *data, int count, char expects)
{
    int i = 0;
    for (i = 0; i < count; i++) {
        if (data[i] != expects) {
            log_err("[%d] %c != %c", i, data[i], expects);
            return 0;
        }
    }

    return 1;
}

int main(int argc, char *argv[])
{
    char from[1003] = { 'a' };
    char to[1003] = { 'c' };
    int rc = 0;

    // setup the from to have some stuff
    memset(from, 'x', 1003);
    // set it to a failure mode
    memset(to, 'y', 1003);
    check(valid_copy(to, 1003, 'y'), "Not initialized right.");

    // use normal copy to 
    rc = normal_copy(from, to, 1003);
    check(rc == 1003, "Normal copy failed: %d", rc);
    check(valid_copy(to, 1003, 'x'), "Normal copy failed.");

    // reset
    memset(to, 'y', 1003);

    // duffs version
    rc = duffs_device(from, to, 1003);
    check(rc == 1003, "Duff's device failed: %d", rc);
    check(valid_copy(to, 1003, 'x'), "Duff's device failed copy.");

    // reset
    memset(to, 'y', 1003);

    // my version
    rc = zeds_device(from, to, 1003);
    check(rc == 1003, "Zed's device failed: %d", rc);
    check(valid_copy(to, 1003, 'x'), "Zed's device failed copy.");

    return 0;
error:
    return 1;
}

```

Remember that this is *bad* code.
It's very interesting though, so struggle with it.

The Analysis

Before you continue, try to figure out what this does.
Consider it a debugging problem.

Clues

* Print this code out so that you can write on some paper.
* Write each of the variables in a table as they
  look when they get initialized right before the ``switch-statement``.
* Follow the logic to the switch, then do the jump to the right case.
* Update the variables, including the ``to``, ``from``, and the
  arrays they point at.

Clues

* When you get to the ``while`` part or my ``goto`` alternative,
  check your variables, and then follow the logic either back to the
  top of the ``do-while`` or to where the ``again`` label is
  located.
* Follow through this manual tracing, updating the variables, until
  you're sure you see how this flows.

Pause!

I will then show you the solution so pause if you do
*NOT* want to see it yet.

Solving It

Watch me walk through how this works to see if it matches what you did.

Extra Credit

* Never use this again.
* Go look at the Wikipedia entry for Duff's device and see if you can
  spot the error.  Read the article, compare it to the version I have here, and try to understand why the Wikipedia code won't work for you
  but worked for Tom Duff.
* Create a set of macros that lets you create any length of device like this.
  For example, what if you wanted to have 32 case statements and didn't want
  to write out all of them? Can you do a macro that lays down eight at a time?

Extra Credit

* Change the ``main`` to conduct some speed tests to see which one is
  really the fastest.
* Read about ``memcpy``, ``memmove``, and ``memset``, and also compare
  their speed.
* Never use this again!

### Exercise 24 Input, Output, Files

The Plan

* Learn the basics of working with files in C.
* Get an initial list of the "f-functions".

The Code

.\ex24\ex24.c

```c
#include <stdio.h>
#include "dbg.h"

#define MAX_DATA 100

typedef enum EyeColor {
    BLUE_EYES, GREEN_EYES, BROWN_EYES,
    BLACK_EYES, OTHER_EYES
} EyeColor;

const char *EYE_COLOR_NAMES[] = {
    "Blue", "Green", "Brown", "Black", "Other"
};

typedef struct Person {
    int age;
    char first_name[MAX_DATA];
    char last_name[MAX_DATA];
    EyeColor eyes;
    float income;
} Person;

int main(int argc, char *argv[])
{
    Person you = {.age = 0 };
    int i = 0;
    char *in = NULL;

    printf("What's your First Name? ");
    in = fgets(you.first_name, MAX_DATA - 1, stdin);
    check(in != NULL, "Failed to read first name.");

    printf("What's your Last Name? ");
    in = fgets(you.last_name, MAX_DATA - 1, stdin);
    check(in != NULL, "Failed to read last name.");

    printf("How old are you? ");
    int rc = fscanf(stdin, "%d", &you.age);
    check(rc > 0, "You have to enter a number.");

    printf("What color are your eyes:\n");
    for (i = 0; i <= OTHER_EYES; i++) {
        printf("%d) %s\n", i + 1, EYE_COLOR_NAMES[i]);
    }
    printf("> ");

    int eyes = -1;
    rc = fscanf(stdin, "%d", &eyes);
    check(rc > 0, "You have to enter a number.");

    you.eyes = eyes - 1;
    check(you.eyes <= OTHER_EYES
            && you.eyes >= 0, "Do it right, that's not an option.");

    printf("How much do you make an hour? ");
    rc = fscanf(stdin, "%f", &you.income);
    check(rc > 0, "Enter a floating point number.");

    printf("----- RESULTS -----\n");

    printf("First Name: %s", you.first_name);
    printf("Last Name: %s", you.last_name);
    printf("Age: %d\n", you.age);
    printf("Eyes: %s\n", EYE_COLOR_NAMES[you.eyes]);
    printf("Income: %f\n", you.income);

    return 0;
error:

    return -1;
}

```

The Analysis

Breaking It

* Trying out *fgets* and the problems with *gets*.
* Feed it */dev/urandom* to give it garbage.

Extra Credit

* Rewrite this to not use ``fscanf`` at all.  You'll need to use
  functions like ``atoi`` to convert the input strings to numbers.
* Change this to use plain ``scanf`` instead of ``fscanf`` to
  see what the difference is.
* Fix it so that their input names get stripped of the trailing newline
  characters and any whites pace.

Extra Credit

* Use ``scanf`` to write a function that reads one character at a time
  and files in the names but doesn't go past the end.  Make this function
  generic so it can take a size for the string, but just make sure you end
  the string with ``'\0'`` no matter what.

### Exercise 25 Variable Argument Functions

The Plan

* Use variable argument functions.
* Write our own simple version of *scanf*.

The Code

.\ex25\ex25.c

```c

#include <stdlib.h>
#include <stdio.h>
#include <stdarg.h>
#include "dbg.h"

#define MAX_DATA 100

int read_string(char **out_string, int max_buffer)
{
    *out_string = calloc(1, max_buffer + 1);
    check_mem(*out_string);

    char *result = fgets(*out_string, max_buffer, stdin);
    check(result != NULL, "Input error.");

    return 0;

error:
    if (*out_string) free(*out_string);
    *out_string = NULL;
    return -1;
}

int read_int(int *out_int)
{
    char *input = NULL;
    int rc = read_string(&input, MAX_DATA);
    check(rc == 0, "Failed to read number.");

    *out_int = atoi(input);

    free(input);
    return 0;

error:
    if (input) free(input);
    return -1;
}

int read_scan(const char *fmt, ...)
{
    int i = 0;
    int rc = 0;
    int *out_int = NULL;
    char *out_char = NULL;
    char **out_string = NULL;
    int max_buffer = 0;

    va_list argp;
    va_start(argp, fmt);

    for (i = 0; fmt[i] != '\0'; i++) {
        if (fmt[i] == '%') {
            i++;
            switch (fmt[i]) {
                case '\0':
                    sentinel("Invalid format, you ended with %%.");
                    break;

                case 'd':
                    out_int = va_arg(argp, int *);
                    rc = read_int(out_int);
                    check(rc == 0, "Failed to read int.");
                    break;

                case 'c':
                    out_char = va_arg(argp, char *);
                    *out_char = fgetc(stdin);
                    break;

                case 's':
                    max_buffer = va_arg(argp, int);
                    out_string = va_arg(argp, char **);
                    rc = read_string(out_string, max_buffer);
                    check(rc == 0, "Failed to read string.");
                    break;

                default:
                    sentinel("Invalid format.");
            }
        } else {
            fgetc(stdin);
        }

        check(!feof(stdin) && !ferror(stdin), "Input error.");
    }

    va_end(argp);
    return 0;

error:
    va_end(argp);
    return -1;
}

int main(int argc, char *argv[])
{
    char *first_name = NULL;
    char initial = ' ';
    char *last_name = NULL;
    int age = 0;

    printf("What's your first name? ");
    int rc = read_scan("%s", MAX_DATA, &first_name);
    check(rc == 0, "Failed first name.");

    printf("What's your initial? ");
    rc = read_scan("%c\n", &initial);
    check(rc == 0, "Failed initial.");

    printf("What's your last name? ");
    rc = read_scan("%s", MAX_DATA, &last_name);
    check(rc == 0, "Failed last name.");

    printf("How old are you? ");
    rc = read_scan("%d", &age);

    printf("---- RESULTS ----\n");
    printf("First Name: %s", first_name);
    printf("Initial: '%c'\n", initial);
    printf("Last Name: %s", last_name);
    printf("Age: %d\n", age);

    free(first_name);
    free(last_name);
    return 0;
error:
    return -1;
}

```

The Analysis

Breaking It

* Change the code so that you forget to pass in the initial size for '%s' formats.
* Give it more data than ``MAX_DATA``, and then see how omitting ``calloc`` in ``read_string`` changes how it works.
* There's a problem where fgets eats the newlines, so try to fix that using
``fgetc`` but leave out the ``\0`` that ends the string.

Extra Credit

* Make double and triple sure that you know what each of the ``out_``
  variables are doing.  Most importantly, you should know what is ``out_string`` is and how it's
  a pointer to a pointer, , so that you understand when you're setting the pointer versus the
  contents is important.  Break down each of the

Extra Credit

* Write a similar function to ``printf`` that uses the varargs system,
  and rewrite ``main`` to use it.
* As usual, read the man page on all of this so that you know what it does
  on your platform.  Some platforms will use macros, others will use
  functions, and some will have these do nothing.  It all depends on the
  compiler and the platform you use.

### Exercise 26 Project logfind

The Plan

Attempt your first project!

logfind

How Projects Work

The projects in this book are designed to make you apply
what you know so far to something "real world".

1. I will tell you when to *pause* so you can try to solve it yourself.
2. You will be given the challenge.  Pause!
3. You will be given clues. Pause!
4. Finally the solution.
5. Then I try to break my own solution.

The Code

logfind.1

.\ex26\logfind.1\logfind.c

```c
#include "dbg.h"

int main(int argc, char *argv[])
{
    check(argc > 2, "USAGE: logfind word word word");

    return 0;

error:
    return 1;
}

```

.\ex26\logfind.1\Makefile

```makefile
CFLAGS=-Wall -g

all: logfind
    ./logfind || true
    ./logfind test test test

```

logfind.2

.\ex26\logfind.2\logfind.c

```c
#include "dbg.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

const size_t MAX_LINE = 1024;

int scan_file(const char *filename, int search_len, char *search_for[])
{
    char *line = calloc(MAX_LINE, 1);
    FILE *file = fopen(filename, "r");
    char *found = NULL;
    int i = 0;

    check_mem(line);
    check(file, "Failed to open file: %s", filename);

    // read each line of the file and search that line for the contents
    while(fgets(line, MAX_LINE-1, file) != NULL && found == NULL) {
        for(i = 0; i < search_len && found == NULL; i++) {
            found = strcasestr(line, search_for[i]);
            if(found) {
                printf("%s\n", filename);
            }
        }
    }

    free(line);
    fclose(file);
    return 0;

error:
    if(line) free(line);
    if(file) fclose(file);

    return -1;
}

int main(int argc, char *argv[])
{
    check(argc > 1, "USAGE: logfind word word word");

    scan_file("logfind.c", argc, argv);

    return 0;

error:
    return 1;
}

```

.\ex26\logfind.2\Makefile

```makefile
CFLAGS=-Wall -g

all: logfind
    ./logfind || true
    ./logfind error

clean:
    rm -f logfind

```

logfind.3

.\ex26\logfind.3\logfind.c

```c
#include "dbg.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <glob.h>

const size_t MAX_LINE = 1024;

int list_files(glob_t *pglob) 
{
    char *line = calloc(MAX_LINE, 1);
    FILE *file = fopen(".logfind", "r");
    int glob_flags = GLOB_TILDE;
    int i = 0;
    int rc = -1;

    check(pglob != NULL, "Invalid glob_t given.");
    check_mem(line);
    check(file, "Failed to open .logfind. Make that first.");

    rc = glob("*.h", glob_flags, NULL, pglob);
    check(rc == 0, "Failed to glob.");
    rc = glob("*.c", glob_flags | GLOB_APPEND, NULL, pglob);
    check(rc == 0, "Failed to glob.");

    for(i = 0; i < pglob->gl_pathc; i++) {
        debug("Matched file: %s", pglob->gl_pathv[i]);
    }

    rc = 0; // all good

error: // fallthrough
    if(line) free(line);
    return rc;
}

int scan_file(const char *filename, int search_len, char *search_for[])
{
    char *line = calloc(MAX_LINE, 1);
    FILE *file = fopen(filename, "r");
    char *found = NULL;
    int i = 0;

    check_mem(line);
    check(file, "Failed to open file: %s", filename);

    // read each line of the file and search that line for the contents
    while(fgets(line, MAX_LINE-1, file) != NULL && found == NULL) {
        for(i = 0; i < search_len && found == NULL; i++) {
            found = strcasestr(line, search_for[i]);
            if(found) {
                printf("%s\n", filename);
            }
        }
    }

    free(line);
    fclose(file);
    return 0;

error:
    if(line) free(line);
    if(file) fclose(file);

    return -1;
}

int main(int argc, char *argv[])
{
    int i = 0;
    glob_t files_found;
    check(argc > 1, "USAGE: logfind word word word");

    check(list_files(&files_found) == 0, "Failed to list files.");

    for(i = 0; i < files_found.gl_pathc; i++) {
        scan_file(files_found.gl_pathv[i], argc, argv);
    }

    globfree(&files_found);
    return 0;

error:
    return 1;
}

```

.\ex26\logfind.3\Makefile

```makefile
CFLAGS=-Wall -g

all: logfind
    ./logfind || true
    ./logfind error

clean:
    rm -f logfind

```

logfind.4

.\ex26\logfind.4\logfind.c

```c
#define NDEBUG
#include "dbg.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <glob.h>

const size_t MAX_LINE = 1024;

int list_files(glob_t *pglob) 
{
    char *line = calloc(MAX_LINE, 1);
    FILE *file = fopen(".logfind", "r");
    int glob_flags = GLOB_TILDE;
    int i = 0;
    int rc = -1;

    check(pglob != NULL, "Invalid glob_t given.");
    check_mem(line);
    check(file, "Failed to open .logfind. Make that first.");

    while(fgets(line, MAX_LINE-1, file) != NULL) {
        line[strlen(line) - 1] = '\0'; // drop the \n ending
        debug("Globbing %s", line);

        rc = glob(line, glob_flags, NULL, pglob);
        check(rc == 0 || rc == GLOB_NOMATCH, "Failed to glob.");

        // dumb work around to a stupid design in glob
        if(glob_flags == GLOB_TILDE) glob_flags |= GLOB_APPEND;
    }

    for(i = 0; i < pglob->gl_pathc; i++) {
        debug("Matched file: %s", pglob->gl_pathv[i]);
    }

    rc = 0; // all good

error: // fallthrough
    if(line) free(line);
    return rc;
}

int scan_file(const char *filename, int search_len, char *search_for[])
{
    char *line = calloc(MAX_LINE, 1);
    FILE *file = fopen(filename, "r");
    char *found = NULL;
    int i = 0;

    check_mem(line);
    check(file, "Failed to open file: %s", filename);

    // read each line of the file and search that line for the contents
    while(fgets(line, MAX_LINE-1, file) != NULL && found == NULL) {
        for(i = 0; i < search_len && found == NULL; i++) {
            found = strcasestr(line, search_for[i]);
            if(found) {
                printf("%s\n", filename);
            }
        }
    }

    free(line);
    fclose(file);
    return 0;

error:
    if(line) free(line);
    if(file) fclose(file);

    return -1;
}

int main(int argc, char *argv[])
{
    int i = 0;
    glob_t files_found;
    check(argc > 1, "USAGE: logfind word word word");

    check(list_files(&files_found) == 0, "Failed to list files.");

    for(i = 0; i < files_found.gl_pathc; i++) {
        scan_file(files_found.gl_pathv[i], argc, argv);
    }

    globfree(&files_found);
    return 0;

error:
    return 1;
}

```

.\ex26\logfind.4\Makefile

```makefile
CFLAGS=-Wall -g

all: logfind
    ./logfind || true
    ./logfind MAX_LINE

clean:
    rm -f logfind

```

logfind.5

.\ex26\logfind.5\logfind.c

```c
#define NDEBUG
#include "dbg.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <glob.h>

const size_t MAX_LINE = 1024;

int list_files(glob_t *pglob) 
{
    char *line = calloc(MAX_LINE, 1);
    FILE *file = fopen(".logfind", "r");
    int glob_flags = GLOB_TILDE;
    int i = 0;
    int rc = -1;

    check(pglob != NULL, "Invalid glob_t given.");
    check_mem(line);
    check(file, "Failed to open .logfind. Make that first.");

    while(fgets(line, MAX_LINE-1, file) != NULL) {
        line[strlen(line) - 1] = '\0'; // drop the \n ending
        debug("Globbing %s", line);

        rc = glob(line, glob_flags, NULL, pglob);
        check(rc == 0 || rc == GLOB_NOMATCH, "Failed to glob.");

        // dumb work around to a stupid design in glob
        if(glob_flags == GLOB_TILDE) glob_flags |= GLOB_APPEND;
    }

    for(i = 0; i < pglob->gl_pathc; i++) {
        debug("Matched file: %s", pglob->gl_pathv[i]);
    }

    rc = 0; // all good

error: // fallthrough
    if(line) free(line);
    return rc;
}

int found_it(int use_or, int found_count, int search_len)
{
    debug("use_or: %d, found_count: %d, search_len: %d", use_or, found_count, search_len);

    if(use_or && found_count > 0) {
        return 1;
    } else if(!use_or && found_count == search_len) {
        return 1;
    } else {
        return 0;
    }
}

int scan_file(const char *filename, int use_or, int search_len, char *search_for[])
{
    char *line = calloc(MAX_LINE, 1);
    FILE *file = fopen(filename, "r");
    int found_count = 0;
    int i = 0;

    check_mem(line);
    check(file, "Failed to open file: %s", filename);

    // read each line of the file and search that line for the contents
    while(fgets(line, MAX_LINE-1, file) != NULL)
    {
        for(i = 0; i < search_len; i++) {
            if(strcasestr(line, search_for[i]) != NULL) {
                debug("file: %s, line: %s, search: %s", filename, line, search_for[i]);
                found_count++;
            }
        }

        if(found_it(use_or, found_count, search_len)) {
            printf("%s\n", filename);
            break;
        } else {
            found_count = 0;
        }
    }

    free(line);
    fclose(file);
    return 0;

error:
    if(line) free(line);
    if(file) fclose(file);

    return -1;
}

int parse_args(int *use_or, int *argc, char **argv[]) 
{
    (*argc)--;
    (*argv)++;

    if(strcmp((*argv)[0], "-o") == 0) {
        *use_or = 1;
        (*argc)--; // skip the -o
        (*argv)++;
        check(*argc > 1, "You need words after -o.");
    } else {
        use_or = 0;
    }

    return 0;
error:
    return -1;
}

int main(int argc, char *argv[])
{
    int i = 0;
    int use_or = 0;
    glob_t files_found;

    check(argc > 1, "USAGE: logfind [-o] words");

    check(parse_args(&use_or, &argc, &argv) == 0, "USAGE: logfind [-o] words");

    check(list_files(&files_found) == 0, "Failed to list files.");

    for(i = 0; i < files_found.gl_pathc; i++) {
        scan_file(files_found.gl_pathv[i], use_or, argc, argv);
    }

    globfree(&files_found);
    return 0;

error:
    return 1;
}

```

.\ex26\logfind.5\Makefile

```makefile
CFLAGS=-Wall -g

all: logfind
    ./logfind || true
    ./logfind MAX_LINE
    ./logfind error MAX LINE
    ./logfind -o error MAX LINE

clean:
    rm -f logfind

```

If you ever get super stuck, you can visit:

To get all of the code for this book.

The Challenge

I want a tool called ``logfind`` that let's me search through log files for
text.  This tool is a specialized version of another tool called ``grep``, but
designed only for log files on a system.

The Challenge

* This tool takes any sequence of words and assumes I mean "and" for them.  So ``logfind zedshaw smart guy`` will find all files that have ``zedshaw`` *and* ``smart`` *and* ``guy`` in them.
* It takes an optional argument of ``-o`` if the parameters are meant to be *or* logic.
* It loads the list of allowed log files from ``~/.logfind``.

The Challenge

* The list of file names can be anything that the ``glob`` function allows.  Refer to ``man 3 glob`` to see how this works.  I suggest starting with just a flat list of exact files, and then add ``glob`` functionality.
* You should output the matching lines as you scan, and try to match them as fast as possible.

Demo

Here is a demo of me using the one I wrote.

Pause!

Now it's time for you to attempt to solve it from just this idea.

The Clues

* Remember to solve it a piece at a time.
* Start with just getting the arguments.
* Then figure out how to open files and just open the ones in this directory.
* Then figure out how to read the files.
* Then find out how to find the arguments in the files.
* Then figure out how glob works.
* Then use glob to find the files and open them.

It helps to do each of these in *main()* then "carve" them out into their
own functions.

Pause!

The Solution

Breaking It

### Exercise 27 Creative and Defensive Programming
 logfind.5 

.\ex27\logfind.5\logfind.c

```c
#define NDEBUG
#include "dbg.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <glob.h>
#include <assert.h>

const size_t MAX_LINE = 1024;

int list_files(glob_t *pglob) 
{
    char *line = calloc(MAX_LINE, 1);
    FILE *file = fopen(".logfind", "r");
    int glob_flags = GLOB_TILDE;
    int i = 0;
    int rc = -1;

    check(pglob != NULL, "Invalid glob_t given.");
    check_mem(line);
    check(file, "Failed to open .logfind. Make that first.");

    while(fgets(line, MAX_LINE-1, file) != NULL) {
        size_t line_length = strnlen(line, MAX_LINE - 1);
        assert(line_length < MAX_LINE && "Got a line length too long.");

        line[line_length] = '\0'; // drop the \n ending
        debug("Globbing %s", line);

        rc = glob(line, glob_flags, NULL, pglob);
        check(rc == 0 || rc == GLOB_NOMATCH, "Failed to glob.");

        // dumb work around to a stupid design in glob
        if(glob_flags == GLOB_TILDE) glob_flags |= GLOB_APPEND;
    }

    for(i = 0; i < pglob->gl_pathc; i++) {
        debug("Matched file: %s", pglob->gl_pathv[i]);
    }

    rc = 0; // all good

error: // fallthrough
    if(line) free(line);
    return rc;
}

int found_it(int use_or, int found_count, int search_len)
{
    debug("use_or: %d, found_count: %d, search_len: %d", use_or, found_count, search_len);

    if(use_or && found_count > 0) {
        return 1;
    } else if(!use_or && found_count == search_len) {
        return 1;
    } else {
        return 0;
    }
}

int scan_file(const char *filename, int use_or, int search_len, char *search_for[])
{
    char *line = calloc(MAX_LINE, 1);
    FILE *file = fopen(filename, "r");
    int found_count = 0;
    int i = 0;

    check_mem(line);
    check(file, "Failed to open file: %s", filename);

    // read each line of the file and search that line for the contents
    while(fgets(line, MAX_LINE-1, file) != NULL)
    {
        for(i = 0; i < search_len; i++) {
            if(strcasestr(line, search_for[i]) != NULL) {
                debug("file: %s, line: %s, search: %s", filename, line, search_for[i]);
                found_count++;
            }
        }

        if(found_it(use_or, found_count, search_len)) {
            printf("%s\n", filename);
            break;
        } else {
            found_count = 0;
        }
    }

    free(line);
    fclose(file);
    return 0;

error:
    if(line) free(line);
    if(file) fclose(file);

    return -1;
}

int parse_args(int *use_or, int *argc, char **argv[]) 
{
    (*argc)--;
    (*argv)++;

    if(strcmp((*argv)[0], "-o") == 0) {
        *use_or = 1;
        (*argc)--; // skip the -o
        (*argv)++;
        check(*argc > 1, "You need words after -o.");
    } else {
        *use_or = 0;
    }

    return 0;
error:
    return -1;
}

int main(int argc, char *argv[])
{
    int i = 0;
    int use_or = 1;
    glob_t files_found;

    check(argc > 1, "USAGE: logfind [-o] words");

    check(parse_args(&use_or, &argc, &argv) == 0, "USAGE: logfind [-o] words");

    check(list_files(&files_found) == 0, "Failed to list files.");

    for(i = 0; i < files_found.gl_pathc; i++) {
        scan_file(files_found.gl_pathv[i], use_or, argc, argv);
    }

    globfree(&files_found);
    return 0;

error:
    return 1;
}

```

.\ex27\logfind.5\Makefile

```makefile
CFLAGS=-Wall -g

all: logfind
    ./logfind || true
    ./logfind MAX_LINE
    ./logfind error MAX LINE
    ./logfind -o error MAX LINE

clean:
    rm -f logfind

```

 logfind.5 

.\ex27\logfind.5\logfind.c

```c
#define NDEBUG
#include "dbg.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <glob.h>
#include <assert.h>

const size_t MAX_LINE = 1024;

int list_files(glob_t *pglob) 
{
    char *line = calloc(MAX_LINE, 1);
    FILE *file = fopen(".logfind", "r");
    int glob_flags = GLOB_TILDE;
    int i = 0;
    int rc = -1;

    check(pglob != NULL, "Invalid glob_t given.");
    check_mem(line);
    check(file, "Failed to open .logfind. Make that first.");

    while(fgets(line, MAX_LINE-1, file) != NULL) {
        size_t line_length = strnlen(line, MAX_LINE - 1);
        assert(line_length < MAX_LINE && "Got a line length too long.");

        line[line_length] = '\0'; // drop the \n ending
        debug("Globbing %s", line);

        rc = glob(line, glob_flags, NULL, pglob);
        check(rc == 0 || rc == GLOB_NOMATCH, "Failed to glob.");

        // dumb work around to a stupid design in glob
        if(glob_flags == GLOB_TILDE) glob_flags |= GLOB_APPEND;
    }

    for(i = 0; i < pglob->gl_pathc; i++) {
        debug("Matched file: %s", pglob->gl_pathv[i]);
    }

    rc = 0; // all good

error: // fallthrough
    if(line) free(line);
    return rc;
}

int found_it(int use_or, int found_count, int search_len)
{
    debug("use_or: %d, found_count: %d, search_len: %d", use_or, found_count, search_len);

    if(use_or && found_count > 0) {
        return 1;
    } else if(!use_or && found_count == search_len) {
        return 1;
    } else {
        return 0;
    }
}

int scan_file(const char *filename, int use_or, int search_len, char *search_for[])
{
    char *line = calloc(MAX_LINE, 1);
    FILE *file = fopen(filename, "r");
    int found_count = 0;
    int i = 0;

    check_mem(line);
    check(file, "Failed to open file: %s", filename);

    // read each line of the file and search that line for the contents
    while(fgets(line, MAX_LINE-1, file) != NULL)
    {
        for(i = 0; i < search_len; i++) {
            if(strcasestr(line, search_for[i]) != NULL) {
                debug("file: %s, line: %s, search: %s", filename, line, search_for[i]);
                found_count++;
            }
        }

        if(found_it(use_or, found_count, search_len)) {
            printf("%s\n", filename);
            break;
        } else {
            found_count = 0;
        }
    }

    free(line);
    fclose(file);
    return 0;

error:
    if(line) free(line);
    if(file) fclose(file);

    return -1;
}

int parse_args(int *use_or, int *argc, char **argv[]) 
{
    (*argc)--;
    (*argv)++;

    if(strcmp((*argv)[0], "-o") == 0) {
        *use_or = 1;
        (*argc)--; // skip the -o
        (*argv)++;
        check(*argc > 1, "You need words after -o.");
    } else {
        *use_or = 0;
    }

    return 0;
error:
    return -1;
}

int main(int argc, char *argv[])
{
    int i = 0;
    int use_or = 1;
    glob_t files_found;

    check(argc > 1, "USAGE: logfind [-o] words");

    check(parse_args(&use_or, &argc, &argv) == 0, "USAGE: logfind [-o] words");

    check(list_files(&files_found) == 0, "Failed to list files.");

    for(i = 0; i < files_found.gl_pathc; i++) {
        scan_file(files_found.gl_pathv[i], use_or, argc, argv);
    }

    globfree(&files_found);
    return 0;

error:
    return 1;
}

```

.\ex27\logfind.5\Makefile

```makefile
CFLAGS=-Wall -g

all: logfind
    ./logfind || true
    ./logfind MAX_LINE
    ./logfind error MAX LINE
    ./logfind -o error MAX LINE

clean:
    rm -f logfind

```

Read The Book

This video is a demonstration of the concepts in the book.

Go read the book.

Demonstration

I will demonstrate each of the following:

* Fail early and openly.
* Document assumptions.
* Prevention over documentation.
* Automate everything.
* Simplify and clarify.
* Question authority.

Fail Early and Openly

Document Assumptions

Prevention over Documentation

Automate Everything

Simplify and Clarify

Question Authority

Bonus: Assume Nothing

### Exercise 28 Intermediate Makefiles

The Plan

* Learn how to create a project skeleton to make starting easier.
* Learn more advanced GNU make tricks.

The Skeleton

.\ex28\c-skeleton

.\ex28\c-skeleton\src\dbg.h

```c
#ifndef __dbg_h__
#define __dbg_h__

#include <stdio.h>
#include <errno.h>
#include <string.h>

#ifdef NDEBUG
#define debug(M, ...)
#else
#define debug(M, ...) fprintf(stderr, "DEBUG %s:%d: " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)
#endif

#define clean_errno() (errno == 0 ? "None" : strerror(errno))

#define log_err(M, ...) fprintf(stderr,\
        "[ERROR] (%s:%d: errno: %s) " M "\n", __FILE__, __LINE__,\
        clean_errno(), ##__VA_ARGS__)

#define log_warn(M, ...) fprintf(stderr,\
        "[WARN] (%s:%d: errno: %s) " M "\n",\
        __FILE__, __LINE__, clean_errno(), ##__VA_ARGS__)

#define log_info(M, ...) fprintf(stderr, "[INFO] (%s:%d) " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)

#define check(A, M, ...) if(!(A)) {\
    log_err(M, ##__VA_ARGS__); errno=0; goto error; }

#define sentinel(M, ...)  { log_err(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#define check_mem(A) check((A), "Out of memory.")

#define check_debug(A, M, ...) if(!(A)) { debug(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#endif

```

.\ex28\c-skeleton\src\libex29.c

```c
#include <stdio.h>
#include <ctype.h>
#include "dbg.h"

int print_a_message(const char *msg)
{
    printf("A STRING: %s\n", msg);

    return 0;
}

int uppercase(const char *msg)
{
    int i = 0;

    // BUG: \0 termination problems
    for(i = 0; msg[i] != '\0'; i++) {
        printf("%c", toupper(msg[i]));
    }

    printf("\n");

    return 0;
}

int lowercase(const char *msg)
{
    int i = 0;

    // BUG: \0 termination problems
    for(i = 0; msg[i] != '\0'; i++) {
        printf("%c", tolower(msg[i]));
    }

    printf("\n");

    return 0;
}

int fail_on_purpose(const char *msg)
{
    return 1;
} 

```

.\ex28\c-skeleton\tests\libex29_tests.c

```c
#include "minunit.h"
#include <dlfcn.h>

typedef int (*lib_function) (const char *data);
char *lib_file = "build/libYOUR_LIBRARY.so";
void *lib = NULL;

int check_function(const char *func_to_run, const char *data,
        int expected)
{
    lib_function func = dlsym(lib, func_to_run);
    check(func != NULL,
            "Did not find %s function in the library %s: %s", func_to_run,
            lib_file, dlerror());

    int rc = func(data);
    check(rc == expected, "Function %s return %d for data: %s",
            func_to_run, rc, data);

    return 1;
error:
    return 0;
}

char *test_dlopen()
{
    lib = dlopen(lib_file, RTLD_NOW);
    mu_assert(lib != NULL, "Failed to open the library to test.");

    return NULL;
}

char *test_functions()
{
    mu_assert(check_function("print_a_message", "Hello", 0),
            "print_a_message failed.");
    mu_assert(check_function("uppercase", "Hello", 0),
            "uppercase failed.");
    mu_assert(check_function("lowercase", "Hello", 0),
            "lowercase failed.");

    return NULL;
}

char *test_failures()
{
    mu_assert(check_function("fail_on_purpose", "Hello", 1),
            "fail_on_purpose should fail.");

    return NULL;
}

char *test_dlclose()
{
    int rc = dlclose(lib);
    mu_assert(rc == 0, "Failed to close lib.");

    return NULL;
}

char *all_tests()
{
    mu_suite_start();

    mu_run_test(test_dlopen);
    mu_run_test(test_functions);
    mu_run_test(test_failures);
    mu_run_test(test_dlclose);

    return NULL;
}

RUN_TESTS(all_tests);

```

The video is probably easier to follow than the book.
Watch me do this.

Using The Skeleton

.\ex28\c-skeleton

.\ex28\c-skeleton\src\dbg.h

```c
#ifndef __dbg_h__
#define __dbg_h__

#include <stdio.h>
#include <errno.h>
#include <string.h>

#ifdef NDEBUG
#define debug(M, ...)
#else
#define debug(M, ...) fprintf(stderr, "DEBUG %s:%d: " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)
#endif

#define clean_errno() (errno == 0 ? "None" : strerror(errno))

#define log_err(M, ...) fprintf(stderr,\
        "[ERROR] (%s:%d: errno: %s) " M "\n", __FILE__, __LINE__,\
        clean_errno(), ##__VA_ARGS__)

#define log_warn(M, ...) fprintf(stderr,\
        "[WARN] (%s:%d: errno: %s) " M "\n",\
        __FILE__, __LINE__, clean_errno(), ##__VA_ARGS__)

#define log_info(M, ...) fprintf(stderr, "[INFO] (%s:%d) " M "\n",\
        __FILE__, __LINE__, ##__VA_ARGS__)

#define check(A, M, ...) if(!(A)) {\
    log_err(M, ##__VA_ARGS__); errno=0; goto error; }

#define sentinel(M, ...)  { log_err(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#define check_mem(A) check((A), "Out of memory.")

#define check_debug(A, M, ...) if(!(A)) { debug(M, ##__VA_ARGS__);\
    errno=0; goto error; }

#endif

```

.\ex28\c-skeleton\src\libex29.c

```c
#include <stdio.h>
#include <ctype.h>
#include "dbg.h"

int print_a_message(const char *msg)
{
    printf("A STRING: %s\n", msg);

    return 0;
}

int uppercase(const char *msg)
{
    int i = 0;

    // BUG: \0 termination problems
    for(i = 0; msg[i] != '\0'; i++) {
        printf("%c", toupper(msg[i]));
    }

    printf("\n");

    return 0;
}

int lowercase(const char *msg)
{
    int i = 0;

    // BUG: \0 termination problems
    for(i = 0; msg[i] != '\0'; i++) {
        printf("%c", tolower(msg[i]));
    }

    printf("\n");

    return 0;
}

int fail_on_purpose(const char *msg)
{
    return 1;
} 

```

.\ex28\c-skeleton\tests\libex29_tests.c

```c
#include "minunit.h"
#include <dlfcn.h>

typedef int (*lib_function) (const char *data);
char *lib_file = "build/libYOUR_LIBRARY.so";
void *lib = NULL;

int check_function(const char *func_to_run, const char *data,
        int expected)
{
    lib_function func = dlsym(lib, func_to_run);
    check(func != NULL,
            "Did not find %s function in the library %s: %s", func_to_run,
            lib_file, dlerror());

    int rc = func(data);
    check(rc == expected, "Function %s return %d for data: %s",
            func_to_run, rc, data);

    return 1;
error:
    return 0;
}

char *test_dlopen()
{
    lib = dlopen(lib_file, RTLD_NOW);
    mu_assert(lib != NULL, "Failed to open the library to test.");

    return NULL;
}

char *test_functions()
{
    mu_assert(check_function("print_a_message", "Hello", 0),
            "print_a_message failed.");
    mu_assert(check_function("uppercase", "Hello", 0),
            "uppercase failed.");
    mu_assert(check_function("lowercase", "Hello", 0),
            "lowercase failed.");

    return NULL;
}

char *test_failures()
{
    mu_assert(check_function("fail_on_purpose", "Hello", 1),
            "fail_on_purpose should fail.");

    return NULL;
}

char *test_dlclose()
{
    int rc = dlclose(lib);
    mu_assert(rc == 0, "Failed to close lib.");

    return NULL;
}

char *all_tests()
{
    mu_suite_start();

    mu_run_test(test_dlopen);
    mu_run_test(test_functions);
    mu_run_test(test_failures);
    mu_run_test(test_dlclose);

    return NULL;
}

RUN_TESTS(all_tests);

```

Now I'll use the skeleton to start a simple project for the next exercise.

The Analysis

Let's look at Makefile in depth.

Extra Credit

### Exercise 29 Libraries and Linking

The Plan

* Learn about libraries and how to link against them.
* Learn how to load a dynamic library from inside C.

The Code

.\ex29\ex29.c

```c
#include <stdio.h>
#include "dbg.h"
#include <dlfcn.h>

typedef int (*lib_function) (const char *data);

int main(int argc, char *argv[])
{
    int rc = 0;
    check(argc == 4, "USAGE: ex29 libex29.so function data");

    char *lib_file = argv[1];
    char *func_to_run = argv[2];
    char *data = argv[3];

    void *lib = dlopen(lib_file, RTLD_NOW);
    check(lib != NULL, "Failed to open the library %s: %s", lib_file,
            dlerror());

    lib_function func = dlsym(lib, func_to_run);
    check(func != NULL,
            "Did not find %s function in the library %s: %s", func_to_run,
            lib_file, dlerror());

    rc = func(data);
    check(rc == 0, "Function %s return %d for data: %s", func_to_run,
            rc, data);

    rc = dlclose(lib);
    check(rc == 0, "Failed to close %s", lib_file);

    return 0;

error:
    return 1;
}

```

.\ex29\libex29.c

```c
#include <stdio.h>
#include <ctype.h>
#include "dbg.h"

int print_a_message(const char *msg)
{
    printf("A STRING: %s\n", msg);

    return 0;
}

int uppercase(const char *msg)
{
    int i = 0;

    // BUG: \0 termination problems
    for(i = 0; msg[i] != '\0'; i++) {
        printf("%c", toupper(msg[i]));
    }

    printf("\n");

    return 0;
}

int lowercase(const char *msg)
{
    int i = 0;

    // BUG: \0 termination problems
    for(i = 0; msg[i] != '\0'; i++) {
        printf("%c", tolower(msg[i]));
    }

    printf("\n");

    return 0;
}

int fail_on_purpose(const char *msg)
{
    return 1;
} 

```

I'll use the project I started from the previous exercise.
This covers some of the extra credit.

The Analysis

This analysis might take a while, but be sure you know Exercise 28 well.

Breaking It

* Wreck the libex29.so file.

Extra Credit

* Were you paying attention to the bad code I have in the ``libex29.c`` functions?
  Do you see how, even though I use a for-loop they still check for ``'\0'``
  endings?  Fix this so that the functions always take a length for the
  string to work with inside the function.
* Read the ``man dlopen`` documentation and read about all of the
  related functions.  Try some of the other options to ``dlopen``
  beside ``RTLD_NOW``.

### Exercise 30 Automated Testing

The Plan

Continue the Exercise 28-29 project and add automated tests to it.

Why Automate Tests

You are a programmer.
Your job is automating.

EVERYTHING

The Code

.\ex30\ex30.c

```c
#include "minunit.h"

char *test_dlopen(int stuff)
{
    return NULL;
}

char *test_functions()
{

    return NULL;
}

char *test_failures()
{

    return NULL;
}

char *test_dlclose()
{

    return NULL;
}

char *all_tests()
{
    mu_suite_start();

    mu_run_test(test_dlopen);
    mu_run_test(test_functions);
    mu_run_test(test_failures);
    mu_run_test(test_dlclose);

    return NULL;
}

RUN_TESTS(all_tests);

```

Adding It To libex29

Breaking It

* Making tests fail first is useful.

Extra Credit

* This works but it's probably a bit messy.  Clean the ``c-skeleton``
  directory up so that it has all of these files, but remove any of the code
  related to Exercise 29.  You should be able to copy this directory
  over and kick-start new projects without much editing.
* Study the ``runtests.sh``, and then go read about ``bash`` syntax
  so you know what it does.  Do you think you could write a C version of this
  script?

### Exercise 31 Common Undefined Behavior

The Plan

Review the issues around Undefined and Unspecified Behavior (UB).

Read The Book

The book lists many of the UB and discusses why they are important to know 
about.

The Code

.\ex31\ex31.c

```c
#include <unistd.h>

int main(int argc, char *argv[])
{
    int i = 0;

    while (i < 100) {
        usleep(3000);
    }

    return 0;
}

```

There is no code for this exercise, just a quick discussion for the book.

Undefined Behavior

* Compiler writers can do whatever they want.
* This means even *nothing*, which will ruin you silently.
* It's best to avoid it.

Unspecified Behavior

* For practical purposes unspecified is the same as undefined.

Handy Tools

* Clang's UB helpful flags.
* Lint tools and static analyzers.

Extra Credit

Spend a day reading through as much of the UB as you can and find examples of each.  Expect lots of frustration and failure when you do this.

<file_sep>#!/bin/bash

echo "Deploying updates to GitHub ..."

# Go To Public folder
cd public

# Add changes to git.
pwd

git config user.name "harryho"
git config user.email "<EMAIL>"

git remote show origin

git checkout master

git pull origin master

## back to root of blog-hugo
cd ..

# Build the project. If using a theme, replace by `hugo -t <yourtheme>`
hugo -t docdock --ignoreCache

# Navigate into public
cd public

git add -A

msg="rebuilding site $(date)"
if [ $# -eq 1 ]; then
  msg="$1"
fi

git commit -m "$msg"
# Push source and build repos.
git push origin master

# Come Back
cd ..
<file_sep>+++
title = "DigitialOcean: First Web Host"
description="UFW, Nginx & Web Host"
weight=3
+++

> Here I contineu to setup SSL certificates for all sites on my web host

## UFW

UFW, or Uncomplicated Firewall, is a front-end to iptables. Its main goal is to make managing your firewall drop-dead simple and to provide an easy-to-use interface.

**DO NOT Enable UFW**

> DO NOT enable UFW without reading through the instructions

### Enable IP V6

* Open the UFW configuration with vi:

```
sudo vi /etc/default/ufw
```

* Make sure "IPV6" is set to "yes", like so:

```
...
IPV6=yes
...
```

### Set default rules

```
sudo ufw deny incoming
sudo ufw allow outgoing
```

### Allow SSH / OpenSSH

* Check app list & enable OpenSSH

```
# List applications
sudo ufw app list

# Allow SSH
sudo ufw allow OpenSSH
```

* Directly allow port 22 or other SSH port, e.g. 2222

```
sudo ufw allow 22 
```


### Enable UFW

```
sudo ufw enable
sudo ufw sattus verbose
```


## Nginx


### Install Nginx

```
sudo apt install ngix
```

### Set UFW

```
# show applications 
sudo ufw app list

# Allow Nginx 
sudo ufw allow 'Nginx Full'
sudo ufw reload
```


## Build Web Host Block

### Create the Directory Structure

* The document root is the directory where the website files for a domain name are stored and served in response to requests. You can set the document root to any location you want.

* Basically, we will create a separate directory for each domain we want to host on our server inside the /var/www directory, which will store the domain website files. 

```
/var/www/
├── domain-one.com
│   └── index.html
```

* Create the root directory domain-one.com:

```
sudo mkdir -p /var/www/domain-one.com
```

* Create an index.html file inside the domain’s root directory.

```bash
sudo touch /var/www/domain-one.com/index.html
```

* Copy following content to the file: __/var/www/domain-one.com/index.html__ 

```html
<!DOCTYPE html>
<html lang="en" dir="ltr">
  <head>
    <meta charset="utf-8">
    <title>domain-one.com </title>
  </head>
  <body>
    <script>
        document.write(
            `<h1>Welecome to domain-one.com  
                 ${new Date().toLocaleString()} 
              </h1>`
        );
    </script>
  </body>
</html>
```

* To avoid any permission issues, change the ownership of the domain document root directory to the Nginx user (www-data):

```bash
sudo chown -R www-data: /var/www/domain-one.com
```

#### Create a Server Block

By default on Ubuntu systems, Nginx server blocks configuration files are stored in __/etc/nginx/sites-available__ directory, which are enabled through symbolic links to the __/etc/nginx/sites-enabled/__ directory.

Open your editor of choice and create the following server block file: __/etc/nginx/sites-available/domain-one.com__

```nginx
server {
    listen 80;
    listen [::]:80;

    root /var/www/domain-one.com;

    index index.html;

    server_name domain-one.com www.domain-one.com;

    access_log /var/log/nginx/domain-one.com.access.log;
    error_log /var/log/nginx/domain-one.com.error.log;

    location / {
        try_files $uri $uri/ =404;
    }
}
```


* To enable the new server block file, create a symbolic link from the file to the sites-enabled directory, which is read by Nginx during startup:

```bash
sudo ln -s /etc/nginx/sites-available/domain-one.com /etc/nginx/sites-enabled/
```

* Test the Nginx configuration for correct syntax:

```
sudo nginx -t
# If there are no errors, the output will look like this:
# nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
# nginx: configuration file /etc/nginx/nginx.conf test is successful

```


* Restart the Nginx service for the changes to take effect

```
sudo systemctl restart nginx
```


### Disable Default Nginx site

* Chanage the default site configuration as below.

```nginx
server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name _;
    deny all;
    return 444;
}
```


### Security

* Next steg is to setup Les's Encrpyt.<file_sep>+++
title = "MySql: DDL & DML"
description="Introduction of SQL: DDL - Data Definition Language DML - Data Manipulation Language"
+++

> As one of most popular open source databases, MySql is mainly used as data storage, aka database. To store the data into MySql server, we need to use SQL - Structural Query Language. But before we store the data to MySql, we need to define the schema which tells MySql how to organize the data in the proper manner. To define the schema, there is a special set of SQL, which we call it DDL - Data Definition Language, such as, CREATE, DROP, ALTER, etc. 

### Create a new database 

```sql
-- Drop the old one  
DROP SCHEMA  IF EXISTS new_dbu

-- Create a new db 
CREATE DATABASE new_db CHARACTER SET utf8 
COLLATE utf8_general_ci;

```

### Create a table

```sql
DROP TABLE IF EXISTS new_table; 

CREATE TABLE new_table (
    id int NOT NULL AUTO_INCREMENT,
    name varchar(50),
    title varchar(50),
    email varchar(250),
    created_date datetime,
    modified_date datetime,
    PRIMARY KEY (id)
);


```

### Query 

- JOIN and INNER JOIN

```sql
SELECT * FROM new_table_a na 
         JOIN new_table_b nb ON nb.new_table_a_id = na.id
         LIMIT 10;

```


### Useful temporary table

```sql
DROP TEMPORARY TABLE IF EXISTS tmp_table; 

CREATE TEMPORARY TABLE tmp_table AS 
SELECT * FROM new_table;

SELECT * FROM tmp_table;

```


### Update data from other table


```sql
UPDATE tableB
INNER JOIN tableA ON tableB.name = tableA.name
SET tableB.value = IF(tableA.value > 0, tableA.value, tableB.value)
WHERE tableA.name = 'Joe'
```


### Delete data

{{% notice tip %}}
How to avoid error: `You can’t specify target table`
{{% /notice %}}

```sql
DELETE FROM TableA
WHERE id NOT IN (
    SELECT * FROM (
        SELECT a.id id FROM TableA a
        JOIN TableB b ON a.tableb_id = b.id
    ) as t
);

```<file_sep>+++
title = "Azure: CAF - 1"
weight = 1
description="Introduction of Cloud Adoption Framework"
+++

## Cloud Adoption Framework

The Microsoft Cloud Adoption Framework for Azure is proven guidance that's designed to help you create and implement the business and technology strategies necessary for your organization to succeed in the cloud. It provides best practices, documentation, and tools that cloud architects, IT professionals, and business decision makers need to successfully achieve short-term and long-term objectives.

The Cloud Adoption Framework brings together cloud adoption best practices from Microsoft employees, partners, and customers. It provides a set of tools, guidance, and narratives that help shape technology, business, and people strategies for driving desired business outcomes during your cloud adoption effort. Review the guidance for each methodology below, providing you with easy access to the right guidance at the right time.

### Structure

methodology |  descripton |
---|----
Strategy| define business justification and expected outcomes of adoption.		
Plan| align actionable adoption plans to business outcomes.
Ready| Prepare the cloud environment for the planned changes.		
Migrate| Migrate and modernize existing workloads.
Innovate| Develop new cloud-native or hybrid solutions.	
Govern| Govern the environment and workloads.
Manage| Operations management for cloud and hybrid solutions.	
Organize| Govern the environment and workloads.


### RACI

RACI - Responsible, Accountable,Consulted,and Informed


### Cloud Adoption Capability

Capability	| Description
----|----
Cloud Adoption	|Deliver technical solutions
Cloud Strategy	|Align technical change to business needs
Cloud Operations	|Support and operate adopted solutions
Cloud Center of Excellence	|Improve quality, speed, and resiliency of adoption
Cloud Governance	|Manage risk, drive consistency, governance, and compliance
Cloud Platform	|Operate and mature the platform
Cloud Automation	| Accelerate adoption and innovation


### SME of Capability

Capability	| Expertise
----|-----
Cloud Strategy  |   Finance
Cloud Strategy	|   Project Management
Cloud Governance|	IT Security
Cloud Governance|	IT Governance
Cloud Platform|	Network
Cloud Platform|	Identity
Cloud Platform|	Virtualization
Cloud Platform|	Disaster Recovery
Cloud Operations|	IT Operations
Cloud Operations|	Application Owner
Cloud Adoption|	Project Lead(s)
Cloud Adoption|	Architect(s)


### Lifecycle


{{<mermaid>}}
graph LR
    S(Define Strategy)
    P(Plan)
    R(Ready)
    A(Adopt)
    subgraph Layer_1
        S-->P
        P-->R
        R-->A
    end
{{</mermaid >}}
{{<mermaid>}}
graph BT
    G(Govern)
    M(Management)
    subgraph Layer_2
        G
        M
    end
{{</mermaid >}}

<file_sep>+++
date = "2018-12-04T14:59:31+11:00"
title = "VPN StrongSwam setup"
description = "VPN with StrongSwam"
+++


## VPN StrongSwan

**strongSwan** is a multiplatform IPsec implementation. The focus of the project is on strong authentication mechanisms using X.509 public key certificates and optional secure storage of private keys and certificates on smartcards through a standardized PKCS#11 interface and on TPM 2.0.

### Launch an instance with Ubuntu

### Update setup script

* Following is setup.sh

```bash
#!/bin/bash
usage() {
  echo "Usage:  strongswan.sh [install|start] [PATADDR] [ETHDEV]

    'install' parameters:
      PATADDR        The private address on MARKETNET (eg. 172.17.133.10)
      ETHDEV         The name of the local ethernet device (eg. etho)
   "
  exit 1
}


install_function () {
  apt update -y
  apt install strongswan -y
  cp ipsec.conf /etc/ipsec.conf
  cp ipsec.secrets /etc/ipsec.secrets
  sysctl -w net.ipv4.ip_forward=1
  ip addr add 172.17.12.127 dev eth0
  iptables -t nat -F
  iptables -t nat -I POSTROUTING -m policy --pol ipsec --dir out -j ACCEPT
  iptables -t nat -A POSTROUTING -d 172.16.17.32/24 -j SNAT --to 172.17.12.127
  iptables-save
}

start_function () {
  ipsec reload
  ipsec rereadsecrets
  ipsec up remote-vpn-b
  ipsec down remote-vpn-b
  ipsec up remote-vpn-a
  ipsec down remote-vpn-a
}

if [ $# -lt 1 ]; then
  echo "No command"
  usage
fi


export operation=$1

if [ "$operation" = "install" ]; then
  install_function
elif [ "$operation" = "start" ]; then
  start_function
fi
```

### Update IPSec config

```
config setup
         strictcrlpolicy=no
         uniqueids = no
         charondebug="ike 3,dmn 0, mgr 3, chd 2, cfg 2, knl 0, net 2, enc 0, esp 3"

conn %default
    auto=route
    compress=no
    type=tunnel
    keyexchange=ikev2
    ike=aes256-sha512-modp2048
    esp=aes256-sha512-modp2048
    leftauth=psk
    rightauth=psk
    authby=secret
    lifetime=28800
    ikelifetime=28800
    rekey=yes
    reauth=no
    inactivity=1800
conn remote-vpn-a
    left=%defaultroute
    leftsubnet=172.17.12.127/32
    leftid=192.168.127.12
    right=172.16.58.3
    rightid=172.16.58.3
    rightsubnet=172.16.17.32/24
conn remote-vpn-b
    left=%defaultroute
    leftsubnet=172.17.12.127/32
    leftid=1172.16.17.32
    right=172.16.17.32
    rightid=172.16.17.32
    rightsubnet=172.16.17.32/24
```

### Update IPSec secrets

```
192.168.127.12 172.16.58.3 : PSK Your_Remote_Key
192.168.127.12 172.16.17.32 : PSK Your_Remote_Key
```


### Setup  & Test StrongSwan

```
sudo bash strongswan.sh install
sudo ipsec reload
sudo ipsec rereadsecrets
sudo ipsec up remote-vpn-b
sudo ipsec down remote-vpn-b
ipsec up remote-vpn-a
sudo ipsec up remote-vpn-a
ipsec up remote-vpn-b
sudo ipsec up remote-vpn-b

```<file_sep>+++
date = "2012-03-24T10:59:31+11:00"
title = "Windows cmd & hotkey - 2"
description="A note for everyone who wants to use Command and Hot Key as hacker "
weight=11
+++

*This article will continue the topic of Windows command & hotkeys. Part-1 shows you common hotkeys and short command lines for `Run` windnow dialog. The rest of this topic will focus on the advanced commands and how to create a batch script with all those commands.*

*Let me clarify something first. Advanced command here does not mean that commands here are very complicated or much more powerful than common ones, which have been shown in the Part-1. Here we call them advanced, because they are used by experienced users to complete their given tasks, and those commands are used seldom by majority people. Comparing with Part-1, advanced commands have some specific features which allow them to do some special jobs, which usually are done by system admin. Advanced command is known as Admin command as well.*


## Advanced commands and usages

### attrib 

* Type `attrib +h a.txt` to hide file and use `attrib -h a.txt` to unhide it. 
* Type `attrib +r a.txt` to change file to read-only and reverse the action by `-r`

### env 

* Type `env>env.txt & notepad env.txt` Display all environment variable in text file

### set 
* Type `set path` to display **PATH** environment variable, which is useful to check if your **PATH** has been setup properly.
* Type `set /P a=b` to set b as value to variable a. It will be used in bat/cmd script. 

### net 
    
**get sub-commands** -- type `net /? `

```
    [ ACCOUNTS | COMPUTER | CONFIG | CONTINUE | FILE | GROUP | HELP |
     HELPMSG | LOCALGROUP | PAUSE | SESSION | SHARE | START | 
     STATISTICS | STOP | TIME | USE | USER | VIEW ]
```

**get sub-command's help** -- type `net [sub-command] /?`

**net view**

* Use `net view` to show a list of computers and network devices on the network.

**net statistics**

* Use `net statistics workstation(/server)` to show the network statistics log for the Server or Workstation service

**net localgroup**

* Use `net localgroup` to show a list of local user group on your computer.

**net user**

* Type `net user %username%` to retrieve your user information 
* Type `net user adminstrator` to check the status of administrator 
* Type `net user administrator /active:yes` to activate adminstrator and inactivate by replacing `yes` with`no`

**net accounts**

* Use `net accounts <user>` to show current user's password and login requirement.
* Use `net accounts <user> /minpwlen:6` to set password minimum length requirement for user.
* Use `net accounts <user> /maxpwage:30` to force user to reset password every 30 days, or use `unlimited` to replace the number `30`, then user's password will never expire.
* User `net accounts /unique:5` to prevent user reuse previous passwords, and default value is 5.


### runas 

```
start command prompt as administrator
runas /user:yourpc\administrator "cmd"

REM ##BE CAREFUL When you try the command below ### 
REM it shows how to create, delete files as admin under C drive root.  
runas /user:yourpc\administrator "cmd /C type \"\">c:\z.txt & \
    dir c:\z.txt & pause & del c:\z.txt " 

```


### sc

* sc command usage: `sc <server> [command] [service name] <option1> <option2>...`

**sc query**

* Basic usage     

    ```
    REM query all service on the PC -- <yourpcname>
    sc \\<yourpcname> query

    REM query status of given service 
    sc query <servicename>
    sc query state= all | find "SERVICE_NAME" 
    ```

* Retrieve service name and state. type parameter can be used twice in some case.
    * state = {active | inactive | all}
    * type = {driver | service | all}
    * type= {own | share | interact | kernel | filesys | rec | adapt}

* __*IMPORTANT*__ 
    * The command options for SC are case sensitive.
    * If you run this inside a batch file, the percent signs (e.g. at %s) need to be doubled.
    * Extra space within option is necessary. e.g. `state= all`

    ```
    REM query all services which are inactive and type are driver and kernel
    sc query state= inactive type= driver type= kernel

    REM get all services name
    for /f "tokens=2" %s in ('sc query state^= all ^| find "SERVICE_NAME"') do @echo %s 

    REM get all services name and state
    for /f "tokens=2" %s in ('sc query state^= all ^| find "SERVICE_NAME"') do @(
        for /f "tokens=4" %t in ('sc query %s ^| find "STATE" ') 
            do @echo %s -- %t
    )
    ```

**sc queryex**

```
REM get all services name and pid
for /f "tokens=2" %s in ('sc queryex state^= all ^| find "SERVICE_NAME"') do @(
    for /f "tokens=3" %t in ('sc queryex %s ^| find "PID" ') 
        do @echo %s -- %t
)

REM get all services name and pid
for /f "tokens=2" %s in ('sc queryex state^= all ^| find "SERVICE_NAME"') do @(
    for /f "tokens=3" %t in ('sc queryex %s ^| find "BINARY_PATH_NAME" ') 
        do @echo %s -- %t
) 
```

**sc qc**

```
REM get all services name and path
for /f "tokens=2" %s in ('sc queryex state^= all ^| find "SERVICE_NAME"') do @(     
    for /f "tokens=3 delims==:" %t in ('sc qc %s ^| find "BINARY_PATH_NAME" ') 
    do @echo %s -- C:%t
)   
```

**sc start/stop**

```
REM start and stop service
sc start  <servicename>

REM query service state
sc query <servicename>

REM stop service
sc stop  <servicename>
```


### ipconfig

* Type `ipconfig /all` to display full configuration information.
* Type `ipconfig /flushdns`    to purge the DNS Resolver cache.


### tasklist

**syntax**

* tasklist[.exe] [/s computer] [/u domain\user [/p password]] [/fo {TABLE|LIST|CSV}] [/nh] [/fi FilterName [/fi FilterName2 [ ... ]]] [/m [ModuleName] | /svc | /v
* FilterName: Status, Imagename,
* Find process by pid

```
REM get the mysqld process info
tasklist /v /fo list /fi "imagename eq mysqld.exe"

REM get the mongod process info
tasklist /v /fo list /fi "imagename eq mongod.exe"

REM get list of running processes under given user  
tasklist /fi "USERNAME ne NT AUTHORITY\SYSTEM" /fi "STATUS eq running"

REM get list of non-responding processes under given user   
tasklist /fi "USERNAME ne NT AUTHORITY\SYSTEM" /fi "STATUS eq not responding" 

REM get process by PID
tasklist /fi "pid eq 4444"
```

### netstat 

* Type `netstat` to get all ports and IP addresses, which are connected or listening 
* Type PID of process which is using some given port, such as 80, 443, 22, etc.

```bash
netstat -ano | find ":80" 
```

* Type the application which is using given port.

```
for /f "tokens=5" %p in ( 'netstat -ano ^| find ":80"') do @(     
    for /f "tokens=1" %s in ( 'tasklist /fi "pid eq %p" ^| find "%p"') do @(
        echo PID:%p -- APP: %s
    ) 
)
```

### taskkill

**syntax**

```
taskkill [/S system [/U username [/P [password]]]]
         { [/FI filter] [/PID processid | /IM imagename] } [/F] [/T]
```

**samples**

```
REM force to stop notepad application and any children processes
taskkill /F /IM notepad.exe /

REM stop process by PID and any children processes
taskkill /PID 1230 /PID 1241 /PID 1253 /T

REM force to stop applications which PID is equal or greater than 10000
REM and windows title of app is not starts with untitle
taskkill /F /FI "PID ge 1000" /FI "WINDOWTITLE ne untitle*"
taskkill /F /FI "USERNAME eq NT AUTHORITY\SYSTEM" /IM notepad.exe
```

### schtasks

* Syntax -- `schtasks /parameter [arguments]`
    * parameters include -- Change, Create, Delete, End, Query, Run, ShowSid  

* Type `schtasks` to list all scheduled tasks

**schtasks /Query**

```
REM get help info                                                                 
SCHTASKS /Query /?

REM query tasks which are scheduled on given system
SCHTASKS /Query /S system /U user /P

REM get list of tasks in details
SCHTASKS /Query /FO LIST /V     

REM get table of running tasks in details and output to csv file 
SCHTASKS /Query /FO TABLE /NH /V | find "Running" >running_tasks.csv
```

## Combination of multiple commands

As we know, usually each command is designed to complete some specific actions, but sometimes we have to combine different commands together to achieve what we want. There are a few ways to put the commands together. 

### Use `&` 

It is used to connect to two commands  and execute them sequentially

* Delete a folder with non-empty subdirectries `test` we need to combine `del` and `rd` together. Actually we can two commands one by one, but we can put it together and just execute once.  

```bash
REM show the folder with non-empty subdirectries
tree test

\path\to\TEST
+---subdir1
|       file1
|       file2
|       
\---subdir2
        file1
        file2

del /s/q test & rd /s/q
```


### Use pipeline `>`

It is used to setup a channel between commands pass the data through the commands. 

Actually you have seen many samples from above advanced commands. I just use a very simple one to show you how it works. 

```
REM write some content to a text file all.txt
echo aaa>all.txt & echo mark aaa >>all.txt & echo mark bbb>>all.txt

```

### Check CPU usage

```
wmic cpu get loadpercentage
@for /f "skip=1" %p in ('wmic cpu get loadpercentage') do @echo %p%

```

### Use `for` 

It is used to loop to combine commands. Please check out the samples for `tasklist` or `netstat`.  


## script

### Basic hello world script

* You can find it on the [home page](https://harryho.github.io)

### Customized script 

* This sample script is used to query temp folders and clean up log files within the folder. 
* We assume you have multiple temp folders in different drives and You want to delete log files inside temp folder and its subdirectries from time to time. Before you delete them, you want to list all files first. You can confirm if you want to delete them or not. 
* Create a file named clean-logs.bat 
* Copy the sample code and tailor anything you want. 
* The sample shows you how to create interative command script and how to combine commands together with the condition statement and loop statement.   

```cmd

@echo off

@echo."Task: "
@echo."You have multiple temp folders in different drives. "
@echo."You want to delete log files inside temp folder and its subdirectries. "
@echo."Before you delete them, you want to list all files first, file list"
@echo."should be sorted by time"

:again 
   echo "Checking all Recycle bins for each drive ..."
   echo.-----------------------
   for /f  %%x in ('wmic logicaldisk get caption  ^| find ":"') do @(
        for /f "tokens=*" %%s in ('tree /f /a %%x\temp ^| find  "log" '  ) do @(
            echo.%%x\temp\%%s
        )
   )

   set /p answer=Do you want to clean up log files (Y/N)?
   if /i "%answer:~,1%" EQU "Y" ( 
       @echo.Y
       goto clean
   )
   if /i "%answer:~,1%" EQU "N" ( 
       @echo.N 
       goto end 
   )  
   echo Please type Y for Yes or N for No
   goto again

:clean
    echo.'deleting logs'
    for /f  %%x in ('wmic logicaldisk get caption  ^| find ":"') do @(
        for /f "tokens=*" %%s in ('tree /f /a %%x\temp ^| find  "log" '  ) do @(
            del "%%x\temp\%%s"
        )
    )

:end
    echo.'exiting program'
```<file_sep>+++

date = "2017-02-15T14:59:31+11:00"
title = "Create a blog site on GitHub Pages"
description="After I setup a blog site with Hugo on my ubuntu machine, I decided to use it to create a blog to GitHub pages on my windows machine "
+++


> If you use Unix-style system, I recommend you to follow the [Hugo Quick Start](https://gohugo.io/overview/quickstart/) and [Hosting on GitHub Pages](https://gohugo.io/tutorials/github-pages-blog) to create a blog to GitHub pages within 5 mins.

> When I decided to use hugo to create a blog on GitHub pages from my windows machine, it took me over 30 mins. I hope this blog can help someone want to do sth similar within Windows environment.*


### Step 1 - Plan and prepare

#### Prerequisite

* You already have *Hugo* on your computer. If not, please follow the instruction to [install hugo on Windows](https://gohugo.io/tutorials/installing-on-windows).

#### Manage your github repositories

* You will have two repositories **blog-hugo** and `<username>.github.io` repositories to hold your hugo content and blog site respectively.

* The **blog-hugo** repository will host actual Hugo’s blog content.

* `<username>.github.io repository` repository will host the static website.

#### Manage your blog site 

* Your *Hugo* blog folder will be "C:\git\blog-hugo" in this example.

* Your blog site will finally sit in C driver and map to repositories as follow

    ```bash
    C:\>
    |--git 
        |--blog-hugo (https://github.com/<yourname>/blog-hugo.git
            |--archetypes
            |--content
            |--data
            |--layouts
            |--public (https://github.com/<yourname>/<yourname>.github.io.git)
            |--themes
            |--

    ```


### Step -2: Create a blog site


#### Create github repositories

* Create on GitHub blog-hugo and <username>.github.io repositories  repository via GitHub website

#### Create a bloodily good blog site 

* Clone blog-hugo via Windows command prompt

    ```bat
    c:\>
    c:\>cd git
    c:\git>git clone <<your-project>-hugo-url> && cd <your-project>-hugo
    ```

* Create hugo site and setup the theme you like

    ```bat   
    C:\git>hugo new site blog-hugo
    C:\git>hugo server -t <yourtheme> -D
    ```

#### Setup a sub module for publish

* Clean up the `public` folder 
* Set submodule inside the blog-hugo and map to folder `public`

    ```bat
    C:\>cd git/blog-hugo
    C:\git>del /s /q  /f  public\*
    C:\git>rd /s /q public
    C:\git>git submodule add -b master https://github.com/<username>/<username>.github.io.git public
    ```

### Deploy to Github page

* Deploy the blog site to GitHub page with the script `deploy.bat`.
* `deploy.bat "Your optional commit message"` will commit the changes to `<username>.github.io`. You can use and tailor the script below as your `deploy.bat`

    ```batch
    @echo OFF

    echo  Deploying updates to GitHub...

    REM Build the project. 
    hugo -t <yourtheme> -D

    REM Go To Public folder
    cd public

    REM Add changes to git.
    git add -A

    REM Commit changes.
    set msg="rebuilding site %date%"
    if  NOT "%1"==""    set msg=%1
    git commit -m '%msg%'

    REM Push source and build repos.
    git push origin master

    REM Come Back
    cd ..
    ```

* You might want to commit the changes to **blog-hugo** repository. Please don't forget to add `public` into the `.gitignore`.

<file_sep>+++
date = "2017-05-04T14:59:31+11:00"
title = "Ubuntu 16 server note"
description = "Ubuntu 16 server note"
draft = false
+++

Prelude

> *This article is mainly to help experienced user install and setup Ubuntu 16 LTS Server. If you are looking for the information for Ubuntu 14, please go to the page [Ubuntu 14 server setup](/os/ubuntu-server-14/)*


## Prerequisites

* You are familiar with Ubuntu, at least you have some experience working on Linux system. 
* You are familiar with basic bash/shell command


## Wireless Setup

> If you install ubuntu server on a laptop, you might end up to setup the wifi first. Usually you won't bring the cable with your laptop wherever you go, also you might just have no cable or run out all cables. Now let's dive into how to setup the wifi on sever. 

* Find out network control installed on your laptop. The column link lists the name of interfaces. The interface name on your laptop will be possible slightly a bit different. It depends on laptop maker. Basically the interface with the type __wlan__ is your wifi interface. 

```bash
networkctl 
```
 
 * You will see network interfaces on your laptop. enp1s0 is the ethernet interface., and `wlp3s0` is your wifi interface

    ```
    IDX LINK           TYPE         OPERATIONAL              SETUP
        1   lo         loopback     n/a                      unmanaged
        2   enp1s0     ether        n/a                      unmanaged
        3   wlp3s0     wlan         n/a                      unmanaged
    ```

* Disable IP V6 if you don't use it. (Highly recommanded for personal laptop user)
 
    - Add  following setting to file  `/etc/sysctl.conf` 

    ```bash
    net.ipv6.conf.all.disable_ipv6 = 1
    net.ipv6.conf.default.disable_ipv6 = 1
    net.ipv6.conf.lo.disable_ipv6 = 1
    ```

    - Reconfigure by running the following command

    ```bash
    sudo sysctl -p

    ## Check the ipv6 status. If you see 1 after running command below, it means ipv6
    ## has been disabled
    cat /proc/sys/net/ipv6/conf/all/disable_ipv6
        
    ```

* Find out the status `wpa_supplicant`

    ```bash
    sudo systemctl status wpa_supplicant

    ## If you find "disabled" in the output, you can simply enable it as below

    sudo systemctl enable wpa_supplicant

    ```

* Find out your wifi ESSID

    ```bash
    sudo iwlist wlp3s0 scan | grep ESSID

    ## If you get error message like "network is down", use ifconfig to bring it up and 
    ## re-run previous commmand    
    sudo ifconfig wlp3s0 up
    ```

* Setup the wpa passphrase for your wifi

   ```bash
   ## Use following command to add pass phrase to your wpa_supplicant
   ## 
   ## wpa_passphrase <your-ESSID> <your-passphrase> | sudo tee /etc/wpa_supplicant.conf

   ## If your ESSID is mywifi and password of wifi is <PASSWORD>, then you will end up the 
   ## command below
   wpa_passphrase mywifi mypasswork | sudo tee /etc/wpa_supplicant.conf

   ```

* Config your wpa supplicant for your wifi interface and run the comand as background process

   ```bash
   sudo wpa_supplicant -c /etc/wpa_supplicant.conf -i wlp3s0 > /dev/null 2>1& &
   ```
   
* Add SSID scan into config `/etc/wpa_supplicant.conf`


    ```bash
    network={
        ssid="mywifi"
        #psk="<PASSWORD>"
        psk=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
        scan_ssid=1
    }
    ```

* Get IP address from external DHCP
  
  ```bash
  sudo dhclient wlp3s0

  ## Check the ip address
  ifconfig wlp3s0
  ```
  
## Wifi Trouble Shooting

* There is no ip address assigned to your wifi interface

    - Check out the wpa_supplicant status
    `sudo systemctl status wpa_supplicant`

    - If you find some error like 'Failed to construct signal', it means
    some network service has been disabled

    `sudo systemctl list-unit-files  --state disabled | grep network`

    - Enable `systemd-networkd.service`, `networking.service` if they are disabled

* The error message comes after the `dhclient`

    - Enable the `squid.service`



* Auto connect to wifi on Starup

```bash
sudo cp /lib/systemd/system/wpa_supplicant.service /etc/systemd/system/wpa_supplicant.service

sudo vi /etc/systemd/system/wpa_supplicant.service
```

* Replace the following line
 `ExecStart=/sbin/wpa_supplicant -u -s -O /run/wpa_supplicant`
 with 
 `ExecStart=/sbin/wpa_supplicant -u -s -c /etc/wpa_supplicant.conf -i wlp3s0`


* Add wifi  interface into auto startup file `/etc/network/interfaces`

```bash
auto lo
iface lo net loopback

auto wlp3s0
iface wlp3s0 net dchp
```




## UFW setup

```bash
sudo ufw enable
sudo ufw allow 80/tcp
sudo ufw allow ssh
sudo ufw allow 443/tcp
sudo ufw allow 8000/tcp
```

## SSH server setup

`!!! For production environment, SSH should be secured by the CA`

```bash
sudo apt-get install openssh-server 

## backup default config 
sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.factory-defaults
sudo chmod a-w /etc/ssh/sshd_config.factory-defaults

## use any editor to update sshd_config 
sudo nano /etc/ssh/sshd_config 

## uncomment  PasswordAuthentication yes to allow remote password login
## Password authentication is only for test environment

## setup ssh auto-start onboot
sudo update-rc.d ssh defaults
```

## !!! Install the software-properties-common Package

```bash 
sudo apt-get install software-properties-common python-software-properties
```


## Time Zone setup

```bash
sudo dpkg-reconfigure tzdata
```


## Install tmux

```bash
sudo apt-get install tmux

```

* Most useful tmux commands

> Ctrl+b " — split pane horizontally.
> 
> Ctrl+b % — split pane vertically.
> 
> Ctrl+b arrow key — switch pane.
> 
> Hold Ctrl+b, don’t release it and hold one of the arrow keys — resize pane.
> 
> Ctrl+b c — (c)reate a new window.
> 
> Ctrl+b , — rename reate a new window.
> 
> Ctrl+b n — move to the (n)ext window.
> 
> Ctrl+b p — move to the (p)revious window.


## Install git 

```bash
sudo add-apt-repository ppa:git-core/ppa
sudo apt-get update
sudo apt-get install git
```



## install docker CE (Ubuntu 16 LTS)

```bash

## Update the apt package index
sudo apt-get update

## Install packages to allow apt to use a repository over HTTPS
sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common

## Add Docker’s official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -



## Verify the last 8 characters of the fingerprint.
sudo apt-key fingerprint xxxxxxxx


##   set up the stable repository
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"

## apt update 
sudo apt-get update

## install docker CE
sudo apt-get install docker-ce

```


## Install JDK 8

* Downlaod the JDK from Oracle website. 

```bash
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer

java -version
```


* Setup environment

```bash
sudo apt-get install oracle-java8-set-default

sudo su
cat >> /etc/environment <<EOL
JAVA_HOME=/usr/lib/jvm/java-8-oracle
JRE_HOME=/usr/lib/jvm/java-8-oracle/jre
EOL
```

* Test JDK with a simple HelloWorld program

    ```java
    import java.util.Calendar;

    class HelloWorld {
        public static void main(String[] args) {
            Calendar cal = Calendar.getInstance();
            int year = cal.get(Calendar.YEAR);
            int month = cal.get(Calendar.MONTH) + 1;
            int day = cal.get(Calendar.DATE);
            int hour = cal.get(Calendar.HOUR_OF_DAY);
            int minute = cal.get(Calendar.MINUTE);
            String username = System.getProperty("user.name");
            System.out.println(username+ ": Hello World! ");
            System.out.println(year + "/" + month + "/" + day + " " + hour + ":" + minute);
        }
    }
    ```

* Compile and run the program

    ```bash
    javac HelloWorld.java
    java HelloWorld.java
    ```


## Install nodejs


* Install Nodejs 8.x

```bash

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs

```

* Install latest npm, yarn and ts

```
sudo npm install -g npm
sudo npm install -g typescript
sudo mpm install -g yarn
```


## Install PHP

* Add new repo 

```bash
sudo apt-get install -y python-software-properties
sudo add-apt-repository -y ppa:ondrej/php
sudo apt-get update -y

apt-cache pkgnames | grep php7.2
```

* Option 1: Install LAMP stack 

```bash 
sudo apt-get install -y apache2
sudo apt-get install -y php7.2 libapache2-mod-php7.2 php7.2-cli php7.2-common \
    php7.2-mbstring php7.2-gd php7.2-intl php7.2-xml php7.2-mysql php7.1-mcrypt php7.2-zip
```

* Option 2: Install LEMP stack

```bash 
sudo apt-get install -y nginx
sudo apt-get install -y php7.2 php7.2-fpm php7.2-cli php7.2-common php7.2-mbstring \
    php7.2-gd php7.2-intl php7.2-xml php7.2-mysql php7.1-mcrypt php7.2-zip
```

* Disable Apache and Nginx if you install both

```bash
sudo systemctl disable apache2.service
sudo systemctl disable nginx.service 
```

## Install Python2, Python3

* Ubuntu has python2 installed by default

```bash
sudo apt-get python-pip 
sudo apt-get install python3-pip
sudo apt-get install python3-dev python-dev

## Install virtualenv 
sudo pip install virtualenv
sudo pip3 install virtualenv
```



## Install Go

* Install Go 

```bash
wget https://storage.googleapis.com/golang/go1.4.linux-amd64.tar.gz


## check hash
shasum -a 256 go*linux-amd64.tar.gz

## install tar ball
sudo tar -C /usr/local -xvzf go1.9.2.linux-amd64.tar.gz
```
* Setup GOROOT by overwriting the file `/etc/environment` with following content

```bash
PATH="/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
JAVA_HOME="/usr/lib/jvm/java-8-oracle"
JRE_HOME="/usr/lib/jvm/java-8-oracle/jre"
GOROOT="/usr/local/go"
```

* Setup GOPATH by adding following lines to the end of `.profile`

```bash
export GOPATH="$HOME/ws/go"
export GOBIN="$GOPATH/bin"
export PATH="$GOPATH/bin:$PATH"
```


* Create a simple `hello.go` file to test 

    ```go
    package main                                                        
                                                                        
    import (                                                            
        "fmt"                                                           
        "log"                                                           
        "os/user"                                                       
    )

    func main(){                                                        
        user, err := user.Current()                                 
        if err != nil {                                             
            log.Fatal(err)                                      
        }                                                                 
        fmt.Printf(user.Name + " said : Hello World! \n" )      
    }                                                                   
    ```

* Run the program

```bash
go run $GOPATH/src/hello.go
go install $GOPATH/src/hello.go
$GOBIN/hello
```



## Install clang & cmake

```bash
sudo apt-get install clang
sudo apt-get install cmake
```

## Install Rust

```bash
$ curl -f -L https://static.rust-lang.org/rustup.sh -O
$ sh rustup.sh
```




<file_sep>+++
date = "2016-02-09T16:56:21+11:00"
title = "Zend Framework 2 MVC Starter"
description="This starter is the starting point of zend framework 2 MVC project"
+++


## Summary

This starter is the starting point of zend framework 2 MVC project. This application is meant to be used as a starting place for those looking to get their feet wet with ZF2. 

## Features

* This starter was built on the zend framework 2.x.
* This starter uses mysql as database setting by default. 
* Include digest authentication by default.
* Include font-awesome files.
* Include Bootstrap 3 without bootstrap-loader.
* Include html5shiiv.js to support older IE browser. 


## Structure of starter

```bash

\path\to\zf2-mvc-starter
+---config                            // Database, authorizaion, authentication setting
+---data          
+---module                            // Customized application sources
|   +---Application                   // Global module used by whole application
|   |   +---config
|   |   |   \---module.config.php     // Register all modules 
|   |   +---language
|   |   +---src
|   |   |   \---Application
|   |   |       +---Controller
|   |   |       \---Factory
|   |   |           \---AuthenticationAdapterFactory.php
|   |   \---view                      // Contains common master, basic layout files 
|   |       +---application
|   |       |   \---index
|   |       +---error
|   |       +---layout
|   |       \---partial
|   +---BookList                      // Customized module for business purpose
|   |   +---config
|   |   +---src
|   |   |   \---BookList
|   |   |       +---Controller
|   |   |       +---Form
|   |   |       \---Model
|   |   \---view
|   |       \---book-list
|   |           \---book
|   \---Test
|       \---config
+---public                          // Contains all fonts, css, images, and js files
|   +---css
|   +---fonts
|   +---img
|   \---js
\---vendor                          // Contains Zend Framework 2 source code
          
    
```


## Screenshot of home page


> ![zf2-mvc-starter](/img/zf2-mvc-starter.png)

## Browse [Repository](https://github.com/harryho/zf2-mvc-starter.git)

<file_sep>
+++


date = "2014-05-04T10:59:31+11:00"
title = "Windows cmd & hotkey - 3"
description="A note for everyone who wants to use Command and Hot Key as hacker "
weight=13
+++



*This article will continue the topic of Windows command & hotkeys. [Part-1](/blog/windows-command-1) shows you common hotkeys and short command lines for `Run` windnow dialog. [Part-2](/os/windows-command-2) advanced commands and how to create a batch script with all those commands. Here I am going to show you another secret weapon in Windows system-VBScript/JScript*

## Breif history

> VBScript/JScript is an Active Scripting language developed by Microsoft that is modeled on Visual Basic. It allows Microsoft Windows system administrators to generate powerful tools for managing computers with error handling, subroutines, and other advanced programming constructs. 

> A VBScript script must be executed within a host environment, of which there are several provided with Microsoft Windows, including: Windows Script Host (WSH), Internet Explorer (IE), and Internet Information Services (IIS). VBScript uses the Component Object Model to access elements of the environment within which it is running.

> JScript is Microsoft's dialect of the ECMAScript standard that is used in Microsoft's Internet Explorer. JScript is implemented as an Active Scripting engine. This means that it can be "plugged in" to OLE Automation applications that support Active Scripting, such as Internet Explorer, Active Server Pages, and Windows Script Host.

> With Cscript.exe, you can run scripts by typing the name of a script file at the command prompt. Like Microsoft Internet Explorer, Windows Script Host serves as a controller of Windows Script compliant scripting engines, but Windows Script Host has very low memory requirements. Windows Script Host is ideal for both interactive and non-interactive scripting needs, such as logon scripting and administrative scripting.


## Sample of VBScript file 

```
Set oFSO = CreateObject("Scripting.FileSystemObject"^)
Set oInput = oFSO.OpenTextFile(WScript.Arguments(0^), 1^)
sData = Replace(oInput.ReadAll, "," ^& VbCrLf, VbCrLf^)
Set oOutput = oFSO.CreateTextFile(WScript.Arguments(1^), True^)
oOutput.Write sData
oInput.Close
oOutput.Close

```

## Replace content script

* Create a file named `repltxt.bat`
* Copy the code into the file `repltxt.bat`
* Run the script


```
@if (@X)==(@Y) @end /* Harmless hybrid line that begins a JScript comment

::************ Documentation ***********
::NEWREPL.BAT version 1.0
:::
:::NEWREPL  Search  replace
:::NEWREPL  /?
:::NEWREPL  /V


@echo off
if .%2 equ . (
      if "%~1" equ "/?" (
        echo."%~f0"
        <"%~f0" cscript //E:JScript //nologo "%~f0" "^:::" "" a
        exit /b 0
      ) else if /i "%~1" equ "/V" (
        <"%~f0" cscript //E:JScript //nologo "%~f0" "^::(NEWREPL\.BAT version)" "$1" a
        exit /b 0
      ) else (
        call :err "Insufficient arguments"
        exit /b 2
      )
)

cscript //E:JScript //nologo "%~f0" %*
exit /b %errorlevel%

:err
>&2 echo ERROR: %~1. Use newrepl /? to get help.
exit /b

************* JScript portion **********/
var rtn=1;
try {
      var env=WScript.CreateObject("WScript.Shell").Environment("Process");
      var args=WScript.Arguments;
      var search=args.Item(0);
      var replace=args.Item(1);
      var options="g";
      if (args.length>2) options+=args.Item(2).toLowerCase();
      var alterations=(options.indexOf("a")>=0);
      if (alterations) options=options.replace(/a/g,"");
      if (options.indexOf("v")>=0) {
          options=options.replace(/v/g,"");
          search=env(search);
          replace=env(replace);
      }
      var search=new RegExp(search,options);
      var str1, str2;

        while (!WScript.StdIn.AtEndOfStream) {
          str1=WScript.StdIn.ReadLine();
          str2=str1.replace(search,replace);
          if (!alterations || str1!=str2) WScript.Stdout.WriteLine(str2);
          if (str1!=str2) rtn=0;
      }
} catch(e) {
      WScript.Stderr.WriteLine("JScript runtime error: "+e.message);
      rtn=3;
}
WScript.Quit(rtn);

```

 The sample above is a bit complicated, and it is combination of Batch script and JScript.
The batch script part checks and validate the input arguments, if there is no issue, then it will trigger the JScript to complete the work.   


## Troubleshooting

From the Windows 7 and on, Windows is no longer to recognize the file with suffix "vbs" as executable file. You need to change the add "vbs" to Windows's registory. <file_sep>+++
date = "2014-03-20T14:59:31+11:00"
title = "JavaScript and OOP"
description="How to power JavaScript with Object Oriendted Programming ... "
+++


## Prerequisites 

* You should have basic knowledge of Javascript and Object Oriented Programming. 
* You should know how to test sample code on Chrome or Firefox. It is simple, just open your browser and click `F12`, copy the code to console and then press `Enter`.   


## What is JavaScript?

* JavaScript, not to be confused with Java, was created in 10 days in May 1995 by <NAME>, then working at Netscape and now of Mozilla. The original name of this language was Mocha, in September of 1995 it was changed to LiveScript, then in December of the same year, the name JavaScript was adopted, because of very popular Java around then.

* JavaScript is the programming language of the web, mobile, back-end API, etc.. It’s one of the most popular and in demand skills in today’s job market for good reason. As a software developer, it is essential that you have a solid understanding of this versatile language.


## What is OOP?

> Object-oriented programming (OOP) is a programming paradigm based on the concept of "objects", which may contain data, in the form of fields, as known as attributes or properties; and actions, in the form of functions, as known as methods.

> For example, Car is an object. The color and model of the car are attributes. Then accelerate to 60km/h, brake to 0km/h, and turn left or right of the car are the actions. From this sample, you can tell OOP makes the code more close to the real world. That is why it is most popular paradigm for developing buisness application.  

## Data types

The JavaScript (ECMAScript) standard defines six data types. Five are primitives, including Boolean, Null, Undefined, Number, String, and Object. In JavaScript, most things are objects, from core JavaScript features like strings and arrays to the browser APIs built on top of JavaScript. You can even create your own objects to encapsulate related functions and variables into efficient packages, and act as handy data containers. The object-oriented nature of JavaScript is important to understand if you want to go further with your knowledge of the language, therefore we've provided this module to help you. Here we teach object theory and syntax in detail, then look at how to create your own objects.


## Object and prototype

***How to define a object***

There are a couple ways to create variable as object. 

```javascript

var obj1 = {};
var obj2 = new Object();
var obj3 = Object.create(null);
console.log( obj );
console.log( obj2 );
console.log( obj3 );

/*
output: 
object {}
object {}
object {}
*/

``` 

Object type gives developers so much power and flexibility to customize their own data type. All JavaScript objects inherit the properties and methods from their prototype. The Object.prototype is on the top of the prototype chain. All JavaScript objects (Date, Array, RegExp, Function, ....) inherit from the Object.prototype.
    
* Object has properties and method. Object's method are the actions that can be performed on objects, they are one of most powerful feature for developers. Let's see how we can create object with properties and methods. 

* Create three cars with basic object usage. 

```javascript

var car1 = { color: 'red', make:'Toyota', model:'Sedan', getInfo: function (){
        console.log( this );
}};
var car2 = { color: 'black', make:'BMW', model:'Coupe', getInfo: function (){
        console.log( this );
}};
var car3 = { color: 'white', make:'Subaru', model:'SUV', getInfo:function (){
        console.log( this );
}};
car1.getInfo();
car2.getInfo(); 
car3.getInfo();

/*
Output:
Object {color: "red", make: "Toyota", model: "Sedan"} 
Object {color: "black", make: "BMW", model: "Coupe"} 
Object {color: "white", make: "Subaru", model: "SUV"}
*/
``` 

* You will find the same method defined in every object. Can we make it better to just define the method once? The answer is Yes. Use an object constructor to create an object prototype. Any new object inherit the same propotype will have the same properties and methods. 

```javascript

var Car = function(color, make, model, getInfo ) {
    this.color='';
    this.make='';
    this.model='';
    this.getInfo= function( time ){
            console.log( this );
    };
};

var car1 = new Car('red','Toyota','Sedan');
var car2 = new Car('black','BMW', 'Coupe');
var car3 = new Car('white','Subaru','SUV');
car1.getInfo();
car2.getInfo(); 
car3.getInfo();
```

* You will get same result as before. If you compare two blocks of code, you may think the second way has more code than the first one. Let's image if you need to create 20 objects and every object with 20 methods, then you totaly need to write 20 X 20 = 400 methods. Object's prototype is powerful, but we need to be careful when we want to use it, especially the `this` prototype. We need discuss this more in detail.      

* Other sample of prototype

```javascript
var Car = function(color, make, model ) {
    this.color='';
    this.make='';
    this.model='';
};
Car.prototype = {        
    getInfo : function( ){
            console.log( this );
    }
};
var car1 = new Car('red','Toyota','Sedan');
var car2 = new Car('black','BMW', 'Coupe');
var car3 = new Car('white','Subaru','SUV');
car1.getInfo();
car2.getInfo(); 
car3.getInfo();
```
* The last way to use prototype is kind of verbose. The second one is more concise and nice is most popular paradigm. 

## Class and inheritance

JavaScript has no built-in way of creating or implementing interfaces.
It also lacks built-in methods for determining whether an object implements the same set of
methods as another object, making it difficult to use objects interchangeably. Luckily, JavaScript
is extremely flexible, making it easy to add these features.

JavaScript has no built-in way of creating or implementing interfaces.
It also lacks built-in methods for determining whether an object implements the same set of
methods as another object, making it difficult to use objects interchangeably. Luckily, JavaScript
is extremely flexible, making it easy to add these features.

**Inheritance issue in Javascript**

```javascript
function Pet() {
    this.name =  "";
    this.species =  "";
    this.offsprings = [];
    this.setName = function ( name ) { this.name = name ;};
    this.deliverBaby = function( obj ){
        this.offsprings.push( obj );
    }
    this.getInfo = function (){
        console.log( " species: ",this.species, " name: " ,this.name );
        console.log( " has ", this.offsprings.length ," offsprings ");
    }
};

function Dog()  {
};

Dog.prototype = new Pet();
Dog.prototype.species = "Dog";

var dog1 = Object.create(new Dog());
dog1.setName ( "Polly");

var dog2 = new Dog();
dog2.setName ( "Lulu");

dog1.deliverBaby( new Dog());
dog2.deliverBaby( new Dog());

dog1.getInfo();
dog2.getInfo();

/*
output :
    species:  Dog  name:  Polly 
    has  2  offsprings              <- It is wrong. It should be 1 only.
    species:  Dog  name:  Lulu 
    has  2  offsprings              <- It is wrong. It should be 1 only.
*/
```

You can tell there is something wrong with the prototype and constructor at a glance. It really confused many developers with C++/Java OOP backgroud. The sample code looks fine, but it doesn't work as other OOP programming language. It is your and <NAME>'s problem, because he was told to make JavaScript look like Java, even there is no built-in OO mechanism at the beginning. This just looks like an odd way of doing class-based OOP without real classes, and leaves the programmer wondering why they didn’t implement proper class-based OOP. JavaScript keeps using constructor, which obscured JavaScript’s true prototypal nature. It turns out most developers don't know how to use it properly and efficiently, including myself at the early stage. 

Function is first-class citizen in JavaScript world, but it’s not really a class. We need to understand the constructor creates an empty object, then sets the prototype of empty object to the prototype property of the constructor, then set constructor function with `this` pointing to the newly-created object, and finally returns the object. You will get more confused after you see this definition. Let's us create a simple sample and take a close look why the constructor and prototype will cause this problem.

```javascript

var MyClass = function(){
    this.name = 'MyClass';
    this.getInfo = function ( ){
        console.log( this );
    }
}
MyClass.prototype.propObject = { id: 0, property: 'property' }

var objectA = new MyClass();
var objectB = new MyClass();

console.log( 'object A:', objectA.name ,  'object B:', objectB.name  );
console.log( 'MyClass.prototype  === objectA.constructor.prototype ? ', 
    MyClass.prototype === objectA.constructor.prototype );
console.log( 'MyClass.prototype  === objectB.constructor.prototype ? ', 
    MyClass.prototype  === objectB.constructor.prototype );

console.log( " objectA.propObject : ", objectA.propObject , 
    " objectB.propObject : ",  objectB.propObject  );

objectA.propObject.id = 1; 
objectA.propObject.property = 'AAA'; 

console.log( " objectA.propObject : ", objectA.propObject,  
    " objectB.propObject : ",  objectB.propObject  );
/*
output :

MyClass object B: MyClass
MyClass.prototype  === objectA.constructor.prototype ?  true
MyClass.prototype  === objectB.constructor.prototype ?  true

objectA.propObject :  Object {id: 0, property: "property"}  
objectB.propObject :  Object {id: 0, property: "property"}

objectA.propObject :  Object {id: 1, property: "AAA"}  
objectB.propObject :  Object {id: 1, property: "AAA"}  
*/
```

If we draw a diagram of above sample, you will see what is happening behind the scene. Since the prototype property is a reference, changing the prototype object’s properties at runtime will affect all objects using the prototype. 

```ini

+------------+             
|  MyClass   |        +---- objectA.prototype
| prototype<----------|
|            |        +---- objectB.prototype
+------------+
```

Now we figure out the root cause. You will say it is easy to fix. We just need to create new prototype for each object, and clone the properties and methods from supper class. Yes, you are right, but it is not I want to recommand to you. First, we need to see if we really inheritance, secondly, if it is better to maintain if use inheritance. 

If we still want to use inheritance, I will suggest not to just inherit the properties, instead of methods. In my opinion, there is very rare of scenario, we really need to inherit method. So we just need to find to proper way to solve the problem of properties inheritance.  


### Object-based Inheritance

```javascript

function Pet(name, master) {
    this.name = name || "";
    this.species = "";
    this.master = master || {
        name: '',
        gender: ''
    };
    this.offsprings = [];
    this.deliverBaby= function ( obj) {
        this.offsprings.push(obj);
    },
    this.getInfo = function () {
        console.log(" species: ", this.species, 
            " name: ", this.name, 
            " master : ", this.master.name, 
            " ", this.master.gender);

        this.offsprings.forEach(function (e) {
            console.log(" has baby :  ", e.name, " ", e.species);
        });
    }
}

function Dog(name, master) {
    Pet.call(this, name, master);
    this.mother = null;
    this.species = "Dog";
}

var dog1 = new Dog('Polly');
dog1.master = {
    name: 'John',
    gender: 'M'
};

var dog2 = new Dog('Lulu', {
    name: 'Ada',
    gender: 'F'
});

dog1.deliverBaby(new Dog('Polly-Baby-Dog'));
dog2.deliverBaby(new Dog('Lulu-Baby-Dog'));
dog2.deliverBaby(new Dog('Lulu-Baby-Dog-2'));

dog1.getInfo();
dog2.getInfo();

/*
output:

Dog  name:  Polly  master :  John   M
has baby :   Polly-Baby-Dog   Dog
species:  Dog  name:  Lulu  master :  Ada   F
has baby :   Lulu-Baby-Dog   Dog
has baby :   Lulu-Baby-Dog-2   Dog
*/

```

After you test, would you ask: "what? how this works? It looks share the same prototype with `this`"? Actually the problem is the special object `this` in Javascript, which is one of the most misunderstood parts of JavaScript. Today it still confuses many other JS developers. If you have experience with other JavaScript framework. You will find many samples which use `that` , `self`, `vm` to replace the built-in `this`. e.g. `var that = {}`, `var self = {}`,etc. Let's see the new version of above sample code. 


```javascript

function Pet(name, master) {
    var self = {};
    self.name = name || "";
    self.species = "";
    self.master = master || {
        name: '',
        gender: ''
    };
    self.offsprings = [];
    return self;
}

function Dog(name, master) {
    var self = {};
    Pet.call(self, name, master);
    self.species = "Dog";
    self.prototype = this.constructor.prototype;
    return self;
}

Dog.prototype = {    
    deliverBaby: function ( self, obj) {
        self.offsprings.push(obj);
    },
    getInfo: function (self) {
        console.log(" species: ", self.species, 
            " name: ", self.name, 
            " master : ", self.master.name, 
            " ", this.master.gender);

        self.offsprings.forEach(function (e) {
            console.log(" has baby :  ", e.name, " ", e.species);
        });
    }
};

var dog1 = new Dog('Polly');
dog1.master = {
    name: 'John',
    gender: 'M'
};
var dog2 = new Dog('Lulu', {
    name: 'Ada',
    gender: 'F'
});

dog1.deliverBaby(dog1, new Dog('Polly-Baby-Dog'));
dog2.deliverBaby(dog2, new Dog('Lulu-Baby-Dog'));
dog2.deliverBaby(dog2, new Dog('Lulu-Baby-Dog-2'));

dog1.getInfo();
dog2.getInfo();

```

Now I rewrite above sample a few lines of code, then you will figour out why it is working, but maybe you still want to implement inheritance as other OOP lanuage C++, Java. Then let's take a look the classical inheritance, which is much more close to other OOP language. In classical inheritance it's impossible (or at least very difficult) to choose which properties you want to inherit. They use virtual base classes and interfaces to solve the diamond problem. It is much more complicated.

### Classical inheritance


```javascript 

function extend(subClass, superClass) {
    var F = function () {};
    F.prototype = superClass.prototype;
    subClass.prototype = new F();
    subClass.prototype.constructor = subClass;
    subClass.superclass = superClass.prototype;

    if (superClass.prototype.constructor == Object.prototype.constructor) {
        superClass.prototype.constructor = superClass;
    }
}

function Pet(name, master) {
    this.name = name || "";
    this.species = "";
    this.master = master || {
        name: '',
        gender: ''
    };
    this.offsprings = [];
}

Pet.prototype.deliverBaby = function (obj) {
    this.offsprings.push(obj);
};

Pet.prototype.getInfo = function () {
    console.log(" species: ", self.species, 
            " name: ", self.name, 
            " master : ", self.master.name, 
            " ", this.master.gender);
    this.offsprings.forEach(function (e) {
        console.log(" has baby :  ", e.name, " ", e.species);
    });
}


function Dog(name, master) {
    Dog.superclass.constructor.call(this, name, master);
    this.species = "Dog";
}

extend(Dog, Pet);

Dog.prototype.getInfo = function () {     
    console.log(" Override --- " );
    Dog.superclass.getInfo.call(this) ;
};

var dog1 = new Dog('Polly');
dog1.master = {
    name: 'John',
    gender: 'M'
};

var dog2 = new Dog('Lulu', {
    name: 'Ada',
    gender: 'F'
});

dog1.deliverBaby(new Dog('Polly-Baby-Dog'));
dog2.deliverBaby(new Dog('Lulu-Baby-Dog'));
dog2.deliverBaby(new Dog('Lulu-Baby-Dog-2'));

dog1.getInfo();
dog2.getInfo();


```

Most programmers who come from a classical background argue that classical inheritance is more powerful than prototypal inheritance. The truth is that prototypal inheritance supports inheriting from multiple prototypes. Prototypal inheritance simply means one object inheriting from another object. 

Whether classical or prototypal, is used to reduce the redundancy in code. Since prototypal inheritance allows for multiple inheritance, code which requires multiple inheritance is less redundant if written using prototypal inheritance rather than in a language which has classical inheritance but no multiple inheritance. 

### Prototypal inheritance 

```javascript

function clone(obj) {
    if (obj === null || typeof obj !== 'object') {
        return obj;
    }

    var temp = obj.constructor(); // give temp the original obj's constructor
    for (var key in obj) {
        temp[key] = clone(obj[key]);
    } 
    return temp;
}

var Pet = {
    name: "",
    species: "",
    master: {
        name: '',
        gender: ''
    },
    offsprings: [],
    deliverBaby: function (obj) {
        this.offsprings.push(obj);
    },
    getInfo: function () {
        console.log(" species: ", this.species, 
        " name: ", name, 
        " master : ", this.master.name,
         " ", this.master.gender);

        this.offsprings.forEach(function (e) {
            console.log(" has baby :  ", e.name, " ", e.species);
        });
    }
};

var Dog = clone(Pet);
Dog.species = 'Dog';

Dog.getInfo = function () {
    console.log(" Override -- species: ", this.species, 
    " name: ", this.name,
    " master : ", this.master.name, 
    " ", this.master.gender);

    this.offsprings.forEach(function (e) {
        console.log(" has baby :  ", e.name, " ", e.species);
    });
};

var dog1 = clone(Dog);
var dog2 = clone(Dog);

dog1.name = 'Polly';
dog1.master = {
    name: 'John',
    gender: 'M'
};
dog2.name = 'Lulu';
dog2.master = {
    name: 'Ada',
    gender: 'F'
};

var dog11 = clone(Dog);
dog11.name = 'Polly-Baby-Dog';
var dog21 = clone(Dog);
var dog22 = clone(Dog);
dog21.name = 'Lulu-Baby-Dog';
dog22.name = 'Lulu-Baby-Dog-2';

dog1.deliverBaby(dog11);
dog2.deliverBaby(dog21);
dog2.deliverBaby(dog22);

dog1.getInfo();
dog2.getInfo();

```

One of the most important advantages of prototypal inheritance is that you can add new properties to prototypes after they are created. This allows you to add new methods to a prototype which will be automatically made available to all the objects which delegate to that prototype.
This allows you to add new methods to a prototype which will be automatically made available to all the objects which delegate to that prototype.This is not possible in classical inheritance because once a class is created you can't modify it at runtime. This is probably the single biggest advantage of prototypal inheritance over classical inheritance, and it should have been at the top.



## Module and namespace

There are quite a lot of benefits from module and namespace, especially when you are going to build some special common api shared within the whole application, even multiple systems across your whole entire enterprise. First thing first, we should not pollute the context, since it will potentially break existing functions or other third party frameworks which have been introduced in your applicatio, vice versa. 

On the other hand, it is a good way to create reusable component, and it is easily for further enhancement, or maybe maintenance. JavaScript is very easy to create a module. One of the most widely used design patterns in JavaScript is the module pattern. 

### Closure

The module pattern makes use of one of the nicer features of JavaScript – closures – in order to give you some control of the privacy of your methods so that third party applications cannot access private data or override it. 

* Simple closure   
  
```javascript
var closureObject = (function() {
    var _privateProperty = 'private';
    var _privateMethod = function () {
        console.log( ' private method ');
    };
    return {
        publicProperty: 'Public Property',
        publicMethod: function() {
            console.log( ' Call ', _privateMethod() , ' from public method ');
        },
        setPrivateProperty: function ( newValue ){
            _privateProperty= newValue;
        },
        getPrivateProperty: function( ){
            return _privateProperty;
        }
    }
}());

console.log(  closureObject.publicProperty );
console.log(  closureObject._privateProperty ); 
// console.log(  closureObject._privateMethod() ); 
// This will cause Uncaught TypeError
console.log(  closureObject.getPrivateProperty() );

closureObject.setPrivateProperty( 'public');
console.log(  closureObject.getPrivateProperty() );

/* 
output:

Public Property
undefined      //--> privateProperty can not be accessed directly
private               
public            //--> privateProperty can be updated by public method
*/
```

From above sample code, you can the JavaScript can easily implement the encapsulation as OOP language. Closure is the base the module pattern, and module is the base of namespace. Maybe you will wonder why we need module and namespace,just closure is good enough for us control the API. If we take a second thought we will realize if some application has the same object called closureObject, both will crash at run time. As a simple solution, we can make a very long, different and ridiculous name to avoid the conflict, but it is not a nice solution. Then module turns out as a better way to solve this problem. 

### Module

Module is not rock science. Actually it is quite easy to implement. 

* Simple module sample

```  javascript
var myModule = (function(undefined) {
    var _privateProperty = 'private';
    var _privateMethod = function () {
        console.log( ' private method ');
    };
    return {
        publicProperty: 'Public Property',
        publicMethod: function() {
            console.log( ' Call ', _privateMethod() , ' from public method ');
        },
        setPrivateProperty: function ( newValue ){
            _privateProperty= newValue;
        },
        getPrivateProperty: function( ){
            return _privateProperty;
        }
    }
}());
```

You may say "What? closure is module." Yes, you can say that. The little difference is the auguements during auto initialization. By having an function argument undefined (the name actually does not matter) which you don't pass a parameter to, you could make sure you have a variable which really is undefined. This technique ensures that it will work as expected, in case it will be excluded to unintential amendment by other script.

Once we create our module, we can simply extend the module with the same technique. 

* Module's extension with override or new api

```javascript
var myModule = (function() {
    ....
}());

var extendModule = (function( m){
     m.publicMethod = function ( newArgument ) { // overload publicMethod 
          //   TODO
     };

     m.newApi = function () {  // 
         // TODO
     };

}(myModule));

```

### Namespace

Now we will go further to namespace, which is based on module technique. Namespace gives you the ability to have public and private properties and methods. 
The code inside doesn’t use the Object Literal notation. Allows you to use $ inside your code without worrying about clashing with other libraries
Allows your library to grow across files using the “window.rtkns = window.rtkns || {}” technique
A common pattern that you will see in many libraries, widgets, and plugins

```javascript

(function (rtkns, $, undefined) {

    rtkns.createNS = function (namespace) {
        var nsparts = namespace.split(".");
        var parent = rtkns;

        if (nsparts[0] === "rtkns") {
            nsparts = nsparts.slice(1);
        }

        for (var i = 0; i < nsparts.length; i++) {
            var partname = nsparts[i];

            if (typeof parent[partname] === "undefined") {
                parent[partname] = {};
            }
            parent = parent[partname];
        }
        return parent;
    };

    var clone = function(obj) {
        if (obj === null || typeof obj !== 'object') {
            return obj;
        }
        // give temp the original obj's constructor
        var temp = obj.constructor(); 
        for (var key in obj) {
            temp[key] = clone(obj[key]);
        } 
        return temp;
    };

    rtkns.clone = clone;

    rtkns.createNS("rtkns");


    rtkns.utils = rtkns.createNS("rtkns.utils");

    rtkns.model = rtkns.createNS("rtkns.model");


    rtkns.model.entity = {
        id: 0,
        createdBy:'',
        modifiedBy:'',
        created: null,
        modified: null,
    };
    var entity = rtkns.model.entity;
    
    rtkns.model.order = clone ( entity);
    var order = rtkns.model.order ;
    order.amount = 0;
    order.description = '';


    rtkns.model.client = clone( entity);
    var client =  rtkns.model.client ;
    client.name = '';
    client.email = '';
    client.orders = [];
    client.purchase = function ( order ){
        this.orders.push( order );
    };

    rtkns.utils.toString = function (entity) {
        return entity?JSON.stringify(entity):entity;
    };


}(window.rtkns = window.rtkns || {}));

var rtkns = window.rtkns;

var client1 = rtkns.clone( rtkns.model.client );
client1.name = 'client 1';
client1.email = '<EMAIL>';
var client2 = rtkns.clone( rtkns.model.client );
client2.name = 'client 2';
client2.email = '<EMAIL>';

var order1 = rtkns.clone( rtkns.model.order );
order1.amount = 100;
order1.description = 'order 1';

var order2 = rtkns.clone( rtkns.model.order );
order2.amount = 600;
order2.description = 'order 2';

client1.purchase( order1 );
client2.purchase( order2 );

console.log(rtkns.utils.toString( client1));
console.log(rtkns.utils.toString( client2));

/*

output:
{"id":0,"createdBy":"","modifiedBy":"","created":null,"modified":null,
    "name":"client 1","email":"<EMAIL>",
    "orders":[{"id":0,"createdBy":"","modifiedBy":"",
        "created":null,"modified":null,
        "amount":100,"description":"order 1"}]}
    
{"id":0,"createdBy":"","modifiedBy":"","created":null,"modified":null,
    "name":"client 2","email":"<EMAIL>",
    "orders":[{"id":0,"createdBy":"","modifiedBy":"",
        "created":null,"modified":null,
        "amount":600,"description":"order 2"}]}
*/
```

The sample above combine namespace and prototypal inheritance. Namespace allows you to add new module for enhancement, and it allows you to organize your API better.  On the other hand, through the globle namespace you can inject customized service, or you can replace it. The disadvantage of namespace, when the source code blows up, it will be a bit more complicated, especially you break different into different files. Mock test or unit test will needs a bit more work to do as well. There is no pattern that is a Silver Bullet, but rather you should assess where you are at and examine the pros and cons of each pattern to address your situation.

## Interfaces

An interface tells programmers what methods a given class implements, which makes it easier to use. Interfaces also stabilize the ways in which different classes can communicate. 

Using any interface implementation in JavaScript will create a small performance hit, due in part to the overhead of having another method invocation. 

The biggest drawback is that there is no way to force other programmers to respect the interfaces you have created. In JavaScript, you must manually ensure that a given class implements an interface. You can mitigate this problem by using coding conventions and helper classes, but it will never entirely go away. Everyone on your project must agree to use them and check for them; otherwise much of their value is lost.

JavaScript does not come with built-in support for interfaces, and there is no Interface keyword, so any method you use to implement this will be very different from what languages such as C++, Java, and making it a little more difficult. JavaScript uses what's called duck typing. (If it walks like a duck, and quacks like a duck, as far as JS cares, it's a duck.) If your object has quack(), walk(), and fly() methods, code can use it wherever it expects an object that can walk, quack, and fly, without requiring the implementation of some "Duckable" interface. 

JavaScript will use Interface object to ensure if the new instance implements the same action as Interface object.  

```javascript


var Interface = function(interfaceName, interfaceMembers) {
    if (!(this instanceof Interface)) {
        return new Interface(interfaceName, interfaceMembers);
    }

    var interfaceObj = this;

    Object.keys(interfaceMembers).forEach(function(memberName) {
        interfaceObj[memberName] = function() {
            Interface.errorDetect(interfaceName, memberName);
        };
    });

    interfaceObj.name = interfaceName;

    return interfaceObj;
};

Interface.errorDetect = function(interfaceName, interfaceMember) {
    throw Error('errorDetect: Class does not implement interface member '
     + interfaceName 
     + '.' 
     + interfaceMember + '()');
};

Interface.ensureImplement = function(obj /*, interfaces */ ) {
    var interfaces = [].slice.call(arguments, 1);

    interfaces.forEach(function(_interface) {
        Object.keys(_interface).forEach(function(interfaceMember) {
            var isFunction = typeof _interface[interfaceMember] === 'function';

            if (isFunction && !obj[interfaceMember]) {
                Interface.errorDetect(_interface.name, interfaceMember);
            }
        });
    });

    return true;
};


```

**How to use this interface**

* Samples below show you how  the Interface can ensure the object implement multiple interfaces.

```javascript
// Sample 1 with only one interface

var ILog = Interface('ILog', {
    logInfo:function(){},
    logWarning:function(){},
    logError:function(){},
}); 


var loggerA = {
    logInfo:function(){},
    logWarning:function(){},
    logError:function(){},
};

// loggerB does not implement all methods
var loggerB = {
    logInfo:function(){},
    logWarning:function(){},
};

console.log(Interface.ensureImplement( loggerA, ILog));
console.log(Interface.ensureImplement( loggerB, ILog));

/*
output:
true
Uncaught Error: errorDetect: Class does not
 implement interface member ILog.logError()
...
*/

// Sample 2 with 2 interfaces

var Submarine = Interface('Submarine', {
    operateUnderwater:function(){}
}); 

var Car = Interface('Car', {
    operateOnRoad:function(){}
}); 

var SubmarineCar = {
    operateUnderwater:function(){},
    operateOnRoad:function(){},
};

console.log(Interface.ensureImplement( SubmarineCar, Submarine, Car ));

/**
output:
true
*/
```









 



<file_sep>+++
title="Good practice - 1"
description="Good practice advice - Part 1"
weight=20
+++

### Strings

#### Change a character

* How to change a character in a string

    ```go
    str:="hello"
    c:=[]byte(s)
    c[0]=’c’
    s2:= string(c) // s2 == "cello"
    ```

#### Substring

* How to take a part(substring) of a string str

```go
substr := str[n:m]
```

#### for-loop

* How to loop over a string str with for or for-range:

```go
// gives only the bytes:
for i:=0; i < len(str); i++ {
     … = str[i]
}
// gives the Unicode characters:
for ix, ch := range str {
    // …
} 
```

#### bytes of str

* Number of bytes in a string str

    ```go
    len(str)
    ```

* Number of characters in a string str

```go
utf8.RuneCountInString(str) // FATEST 
len([]int(str))
```

#### Concat strings

* Best performance with byte buffer

    ```go
    var buffer bytes.Buffer
    for {
        if s, ok := getNextString(); ok { //method getNextString() not shown here
            buffer.WriteString(s)
        } else {
            break
        }
    }
    ```
* Simple way 

```go
strings.Join()
```

#### Command-line args

* Use flag and os package


### Array

#### Cut the last element 

* How to cut the last element of an array or slice line

```go
line = line[:len(line)-1]
```

#### Loop over the array

* for-loop

```go
for i:=0; i < len(arr); i++ {
    … = arr[i]
}
```

* for range

```go
for ix, value := range arr {
    …
}
```

#### Search value in 2D array

```go
found := false
Found: for row := range arr2Dim {
    for column := range arr2Dim[row] {
        if arr2Dim[row][column] == V {
            found = true
            break Found
        }
    }
}
```

####  Reverse 

```go
func ReverseInts(s []int) {
    first := 0
    last := len(s) - 1
    for first < last {
        s[first], s[last] = s[last], s[first]
        first++
        last--
    }
}
```


### Maps

#### loop with range

```go
for key, value := range map1 {
… 
}
```

#### Test if key exists

```go
val1, isPresent = map1[key1]
// which gives: val or zero-value, true or false
```

#### Deleting a key in a map

```go
delete(map1, key1)
```


### Interface

#### Test if a value implements Stringer

```go
if v, ok := v.(Stringer); ok {
    fmt.Printf("implements String(): %s\n", v.String());
}
```

####  A type classifier:

```go
func classifier(items ...interface{}) {
    for i, x := range items {
        switch x.(type) {
            case bool: fmt.Printf("param #%d is a bool\n", i)
            case float64: fmt.Printf("param #%d is a float64\n", i)
            case int, int64: fmt.Printf("param #%d is an int\n", i)
            case nil: fmt.Printf("param #%d is nil\n", i)
            case string: fmt.Printf("param #%d is a string\n", i)
            default: fmt.Printf("param #%d’s type is unknown\n", i)
        }
    }
}
```


















<file_sep>+++
title="Good practice - 2"
description="Good practice advice - Part 2"
weight=21
+++




### Goroutines and channels

* Performance advice:

> A rule of thumb if you use parallelism to gain efficiency over serial computation: the amount of work done inside goroutine has to be much higher than the costs associated with creating goroutines and sending data back and forth between them.


#### Using buffered channels

* Using buffered channels for performance:

    A buffered channel can easily double its throughput, depending on the context the performance gain can be 10x or more. You can further try to optimize by adjusting the capacity of the channel.

#### Limiting the number of items in a channel

* Limiting the number of items in a channel and packing them in arrays:
Channels become a bottleneck if you pass a lot of individual items through them. You can work around this by packing chunks of data into arrays and then unpacking on the other end. This can be a speed gain of a factor 10x.


#### loop over a channel

* How to loop over a channel ch with a for—range:

    ```go
    for v := range ch {
        // do something with v
    }
    ```

#### Test a channel is closed 

* How to test if a channel ch is closed:

    ```go
    //read channel until it closes or error-condition
    for {
        if input, open := <-ch; !open {
        break
        }
        fmt.Printf(“%s “, input)
    }
    ```

#### Samaphore pattern

*  How to use a channel to let the main program wait until the goroutine completes?
(Semaphore pattern) 

    ```go

    ch := make(chan int) // Allocate a channel.
    // Start something in a goroutine; when it completes, signal on the channel.
    go func() {
    // doSomething
    ch <- 1 // Send a signal; value does not matter.
    }()
    doSomethingElseForAWhile()
    <-ch // Wait for goroutine to finish; discard sent value.
    // If the routine must block forever, omit ch <- 1 from the lambda function.
    ```

#### Channel Factory pattern

* Channel Factory pattern: the function is a channel factory and starts a lambda
function as goroutine populating the channel

    ```go
    func pump() chan int {
        ch := make(chan int)
        go func() {
            for i := 0; ; i++ {
                ch <- i
            }
        }()
        return ch
    }
    ```

#### Channel Iterator pattern

* Channel Iterator pattern: Implement the Iter() method of a container returns a channel for the calling for-loop to read from.

    ```go
    func (c *container) Iter() <-chan items {
        ch := make(chan item)
        go func() {
            for i := 0; i < c.Len(); i++ {
                // or use a for-range loop
                ch <- c.items[i]
            }
        }()
        return ch
    }

    // The code which calls this method can then iterate over the container
    for x := range container.Iter() { ... }
    ```


#### Limiting the number of requests 

* Limiting the number of requests processed concurrently

    ```go
    const (
        AvailableMemory         = 10 << 20                                  // 10 MB, for example
        AverageMemoryPerRequest = 10 << 10                                  // 10 KB
        MAXREQS                 = AvailableMemory / AverageMemoryPerRequest // here amounts to 1000
    )

    var sem = make(chan int, MAXREQS)

    type Request struct {
        a, b   int
        replyc chan int
    }

    func process(r *Request) {
        // Do something
        // May take a long time and use a lot of memory or CPU
    }
    func handle(r *Request) {
        process(r)
        // signal done: enable next request to start
        // by making 1 empty place in the buffer
        <-sem
    }
    func Server(queue chan *Request) {
        for {
            sem <- 1
            // blocks when channel is full (1000 requests are active)
            // so wait here until there is capacity to process a request
            // (doesn’t matter what we put in it)
            request := <-queue
            go handle(request)
        }
    }

    func main() {
        fmt.Println(" AvailableMemory ", AvailableMemory)
        fmt.Println(" AverageMemoryPerRequest ", AverageMemoryPerRequest)
        queue := make(chan *Request)
        go Server(queue)
    }
    ```



#### Parallelling computing over a few cores


* Parallelling a computing over a numbers of CPU cores

```go
const NCPU = 4

func DoAll() {
    sem := make(chan int, NCPU) // Buffering optional but sensible.
    for i := 0; i < NCPU; i++ {
        go DoPart(sem)

    }
    // Drain the channel sem, waiting for NCPU tasks to complete
    for i := 0; i < NCPU; i++ {
        <-sem // wait for one task to complete
    }
    // All done.
}
func DoPart(sem chan int) {
    // do the part of the computation

    sem <- 1 // signal that this piece is done
}

func main() {
    runtime.GOMAXPROCS(NCPU)
    DoAll()
}
```



#### Paralleling computing over a large amount of data

```go
func ParallelProcessData (in <- chan *Data, out <- chan *Data) {
    // make channels:
    preOut := make(chan *Data, 100)
    stepAOut := make(chan *Data, 100)
    stepBOut := make(chan *Data, 100)
    stepCOut := make(chan *Data, 100)
    // start parallel computations:
    go PreprocessData(in, preOut)
    go ProcessStepA(preOut, stepAOut)
    go ProcessStepB(stepAOut, stepBOut)
    go ProcessStepC(stepBOut, stepCOut)
    go PostProcessData(stepCOut, out
}
```

#### Simple timeout pattern

```go
timeout := make(chan bool, 1)
go func() {
    time.Sleep(1e9) // one second
    timeout <- true
}()
    
select {
    case <-ch:
    // a read from ch has occurred
    case <-timeout:
    // the read from ch has timed out
}
```
#### Use in- & out- channel instead of lock

```go
func Worker(in, out chan *Task) {
    for {
        t := <-in
        process(t)
        out <- t
    }
}
```


#### Concurrent access to object

```go
type Person struct {
    Name   string
    salary float64
    chF    chan func()
}

func NewPerson(name string, salary float64) *Person {
    p := &Person{name, salary, make(chan func())}
    go p.backend()
    return p
}
func (p *Person) backend() {
    for f := range p.chF {
        f()
    }
}

// Set salary.
func (p *Person) SetSalary(sal float64) {
    p.chF <- func() { p.salary = sal }
}

// Retrieve salary.
func (p *Person) Salary() float64 {
    fChan := make(chan float64)
    p.chF <- func() { fChan <- p.salary }
    return <-fChan
}
func (p *Person) String() string {
    return "Person - name is: " + p.Name + " - salary is: " + strconv.
        FormatFloat(p.Salary(), 'f', 2, 64)
}
func main() {
    bs := NewPerson("<NAME>", 2500.5)
    fmt.Println(bs)
    bs.SetSalary(4000.25)
    fmt.Println("Salary changed:")
    fmt.Println(bs)
}
/* Output Person - name is: <NAME> - salary is: 2500.50
Salary changed:
Person - name is: <NAME> - salary is: 4000.25 *
*/
```

#### Abandon synchronous calls

* Abandon synchronous calls that run too long

    ```go
    ch := make(chan error, 1)
    go func() { ch <- client.Call(“Service.Method”, args, &reply) } ()
    select {
        case resp := <-ch:
        // use resp and reply
        case <-time.After(timeoutNs):
        // call timed out
        break
    }
    ```

#### Benchmarking goroutines

```go
func main() {
    fmt.Println("sync", testing.Benchmark(BenchmarkChannelSync).String())
    fmt.Println("buffered", testing.Benchmark(BenchmarkChannelBuffered).String())
}

func BenchmarkChannelSync(b *testing.B) {
    ch := make(chan int)
    go func() {
        for i := 0; i < b.N; i++ {
            ch <- i
        }
        close(ch)
    }()
    for _ = range ch {
    }
}

func BenchmarkChannelBuffered(b *testing.B) {
    ch := make(chan int, 128)
    go func() {
        for i := 0; i < b.N; i++ {
            ch <- i
        }
        close(ch)
    }()
    for _ = range ch {
    }
}

/* Output:
sync  3000000           420 ns/op
buffered 10000000           103 ns/op
*/
```

#### Stopping a goroutine

```go
runtime.Goexit()
```



<file_sep>+++
title = "JS & ES snippet"
description="JavaScript, ECMAScript snippets"
+++

### Setup the global node modules

* Add an environment variable to tell node where the global node module sits

```bash
# I use nvm to manage my node and node modules
export NODE_PATH=${HOME}/.nvm/versions/node/<node_version>/lib/node_modules
```

* If you use npm by default, you may have permission problem to access the node modules. I suggest you to set a customized global node module folder under your home directory. 

    * Create a new folder under your home directory
    * Install npm to new global node module
    * update environment variables in the profile

    ```bash
    mkdir $HOME/.node_modules
    npm config set prefix $HOME/.node_modules
    npm install -g npm
    echo 'export NODE_MODULES=$HOME/.node_modules' >> $HOME/.profile
    echo 'export PATH=$PATH:$NODE_MODULES/bin' >> $HOME/.profile
    source $HOME/.profile
    ```


### Parsing CSV file

#### Parse csv file with fast-csv

* Install package fast-csv globally

    npm install -g fast-csv

* Assume there is a csv file named test.csv, which contains a few contacts

##### Sample 1 

```js
const fs = require("fs")
const csv = require('fast-csv');

fs.createReadStream('test.csv')
    .pipe(csv.parse({headers:true}))
    .on('error', error => console.error(error))
    .on('data', row => console.log(`ROW=${JSON.stringify(row)}`))
    .on('end', rowCount => console.log(`Parsed ${rowCount} rows`));


```


##### Sample 2

* Validate empty field 

```js
const csv = require('fast-csv')
const fs = require('fs')
const fileStream = fs.createReadStream('test.csv')
const parser = csv.parse({ ignoreEmpty: true, headers: true, trim: true })

const invalidFilter

fileStream
  .pipe(parser)
  .validate(data => data.last_name && data.mobile.startsWith('04'))
  .on('error', error => console.error(error))
  .on('data', row => console.log(`Valid [row=${JSON.stringify(row)}]`))
  .on('data-invalid', (row, rowNumber) => console.log(`Invalid [rowNumber=${rowNumber}] [row=${JSON.stringify(row)}]`))
  .on('end', rowCount => console.log(`Parsed ${rowCount} rows`))
```



<file_sep>+++
title = "AWS: S3 - 1"
description = "S3 Part 1 - Storage"
weight=7
+++

## S3 Part 1

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. 

### Storage feature

Amazon S3 has various features you can use to organize and manage your data in ways that support specific use cases, enable cost efficiencies, enforce security, and meet compliance requirements. Data is stored as objects within resources called “buckets”, and a single object can be up to 5 terabytes in size. Amazon S3 offers a range of storage classes designed for different use cases. 

### S3 Standard


S3 Standard offers high durability, availability, and performance object storage for frequently accessed data. Because it delivers low latency and high throughput, S3 Standard is appropriate for a wide variety of use cases, including cloud applications, dynamic websites, content distribution, mobile and gaming applications, and big data analytics. 



### S3 Standard-IA (Infrequent Access)

S3 Standard-IA is for data that is accessed less frequently, but requires rapid access when needed. S3 Standard-IA offers the high durability, high throughput, and low latency of S3 Standard, with a low per GB storage price and per GB retrieval fee. This combination of low cost and high performance make S3 Standard-IA ideal for long-term storage, backups, and as a data store for disaster recovery files.



### S3 One Zone-IA (Infrequent Access)

S3 One Zone-IA is for data that is accessed less frequently, but requires rapid access when needed. Unlike other S3 Storage Classes which store data in a minimum of three Availability Zones (AZs), S3 One Zone-IA stores data in a single AZ and costs 20% less than S3 Standard-IA. S3 One Zone-IA is ideal for customers who want a lower-cost option for infrequently accessed data but do not require the availability and resilience of S3 Standard or S3 Standard-IA. It’s a good choice for storing secondary backup copies of on-premises data or easily re-creatable data. 


### S3 Glacier

S3 Glacier is a secure, durable, and low-cost storage class for data archiving. You can reliably store any amount of data at costs that are competitive with or cheaper than on-premises solutions. To keep costs low yet suitable for varying needs, S3 Glacier provides three retrieval options that range from a few minutes to hours.


### S3 Glacier Deep Archive

S3 Glacier Deep Archive is Amazon S3’s lowest-cost storage class and supports long-term retention and digital preservation for data that may be accessed once or twice in a year. It is designed for customers — particularly those in highly-regulated industries, such as the Financial Services, Healthcare, and Public Sectors — that retain data sets for 7-10 years or longer to meet regulatory compliance requirements. 







