text:
Q: Setting up a forward proxy using Apache Traffic Server which forwards all request to another proxy and caches responses Setting up a forward proxy using Apache Traffic Server which forwards all request to another proxy and caches responses. I want to setup caching in front of my crawlers so that unwanted js and CSS requests can be cached all the time. I already have a setup where a load balancer is forwarding proxy requests to the setup of proxy services. I want to put ATS between the crawlers and load balancer. My current ATS records.config and parent.config are given below.
Currently when I try to do this,  curl --proxy localhost:8080 "https://exmaple.com/path" I get this error : curl: (56) Received HTTP code 502 from proxy after CONNECT
Here is my parent.config
dest_domains=. parent="localhost:3128;" round_robin=strict
Here is my records.config
##############################################################################
# *NOTE*: All options covered in this file should be documented in the docs:
#
#    https://docs.trafficserver.apache.org/en/latest/reference/configuration/records.config.en.html
##############################################################################
##############################################################################
# Thread configurations. Docs:
#    https://docs.trafficserver.apache.org/records.config#proxy-config-exec-thread-autoconfig
##############################################################################
CONFIG proxy.config.exec_thread.autoconfig INT 1
CONFIG proxy.config.exec_thread.autoconfig.scale FLOAT 1.500000
CONFIG proxy.config.exec_thread.limit INT 2
CONFIG proxy.config.ssl.number.threads INT 0
CONFIG proxy.config.accept_threads INT 1
CONFIG proxy.config.task_threads INT 2
CONFIG proxy.config.cache.threads_per_disk INT 8
    # https://docs.trafficserver.apache.org/records.config#proxy-config-exec-thread-affinity
CONFIG proxy.config.exec_thread.affinity INT 0
##############################################################################
# Specify server addresses and ports to bind for HTTP and HTTPS. Docs:
#    https://docs.trafficserver.apache.org/records.config#proxy-config-http-server-ports
##############################################################################
CONFIG proxy.config.http.server_ports STRING 8080
##############################################################################
# Via: headers. Docs:
#     https://docs.trafficserver.apache.org/records.config#proxy-config-http-insert-response-via-str
##############################################################################
CONFIG proxy.config.http.insert_request_via_str INT 1
CONFIG proxy.config.http.insert_response_via_str INT 0
##############################################################################
# Parent proxy configuration, in addition to these settings also see parent.config. Docs:
#    https://docs.trafficserver.apache.org/records.config#parent-proxy-configuration
#    https://docs.trafficserver.apache.org/en/latest/reference/configuration/parent.config.en.html
##############################################################################
CONFIG proxy.config.http.parent_proxy_routing_enable INT 1
CONFIG proxy.config.http.parent_proxy.retry_time INT 300
CONFIG proxy.config.http.parent_proxy.connect_attempts_timeout INT 30
CONFIG proxy.config.http.forward.proxy_auth_to_parent INT 0
CONFIG proxy.config.http.uncacheable_requests_bypass_parent INT 1
##############################################################################
# HTTP connection timeouts (secs). Docs:
#    https://docs.trafficserver.apache.org/records.config#http-connection-timeouts
##############################################################################
CONFIG proxy.config.http.keep_alive_no_activity_timeout_in INT 115
CONFIG proxy.config.http.keep_alive_no_activity_timeout_out INT 120
CONFIG proxy.config.http.transaction_no_activity_timeout_in INT 30
CONFIG proxy.config.http.transaction_no_activity_timeout_out INT 30
CONFIG proxy.config.http.transaction_active_timeout_in INT 900
CONFIG proxy.config.http.transaction_active_timeout_out INT 0
CONFIG proxy.config.http.accept_no_activity_timeout INT 120
CONFIG proxy.config.net.default_inactivity_timeout INT 86400
##############################################################################
# Origin server connect attempts. Docs:
#    https://docs.trafficserver.apache.org/records.config#origin-server-connect-attempts
##############################################################################
CONFIG proxy.config.http.connect_attempts_max_retries INT 6
CONFIG proxy.config.http.connect_attempts_max_retries_dead_server INT 3
CONFIG proxy.config.http.connect_attempts_rr_retries INT 3
CONFIG proxy.config.http.connect_attempts_timeout INT 30
CONFIG proxy.config.http.post_connect_attempts_timeout INT 1800
CONFIG proxy.config.http.down_server.cache_time INT 300
CONFIG proxy.config.http.down_server.abort_threshold INT 10
##############################################################################
# Negative response caching, for redirects and errors. Docs:
#    https://docs.trafficserver.apache.org/records.config#negative-response-caching
##############################################################################
CONFIG proxy.config.http.negative_caching_enabled INT 0
CONFIG proxy.config.http.negative_caching_lifetime INT 1800
##############################################################################
# Proxy users variables. Docs:
#    https://docs.trafficserver.apache.org/records.config#proxy-user-variables
##############################################################################
CONFIG proxy.config.http.anonymize_insert_client_ip INT 1
CONFIG proxy.config.http.insert_squid_x_forwarded_for INT 1
##############################################################################
# Security. Docs:
#    https://docs.trafficserver.apache.org/records.config#security
##############################################################################
CONFIG proxy.config.http.push_method_enabled INT 0
##############################################################################
# Enable / disable HTTP caching. Useful for testing, but also as an
# overridable (per remap) config
##############################################################################
CONFIG proxy.config.http.cache.http INT 1
##############################################################################
# Cache control. Docs:
#    https://docs.trafficserver.apache.org/records.config#cache-control
#    https://docs.trafficserver.apache.org/en/latest/reference/configuration/cache.config.en.html
##############################################################################
CONFIG proxy.config.http.cache.ignore_client_cc_max_age INT 1
CONFIG proxy.config.http.normalize_ae_gzip INT 1
CONFIG proxy.config.http.cache.cache_responses_to_cookies INT 1
CONFIG proxy.config.http.cache.cache_urls_that_look_dynamic INT 1
    # https://docs.trafficserver.apache.org/records.config#proxy-config-http-cache-when-to-revalidate
CONFIG proxy.config.http.cache.when_to_revalidate INT 0
    # https://docs.trafficserver.apache.org/records.config#proxy-config-http-cache-required-headers
CONFIG proxy.config.http.cache.required_headers INT 0
##############################################################################
# Heuristic cache expiration. Docs:
#    https://docs.trafficserver.apache.org/records.config#heuristic-expiration
##############################################################################
CONFIG proxy.config.http.cache.heuristic_min_lifetime INT 3600
CONFIG proxy.config.http.cache.heuristic_max_lifetime INT 86400
CONFIG proxy.config.http.cache.heuristic_lm_factor FLOAT 0.100000
##############################################################################
# Network. Docs:
#    https://docs.trafficserver.apache.org/records.config#network
##############################################################################
CONFIG proxy.config.net.connections_throttle INT 30000
##############################################################################
# RAM and disk cache configurations. Docs:
#    https://docs.trafficserver.apache.org/records.config#ram-cache
#    https://docs.trafficserver.apache.org/en/latest/reference/configuration/storage.config.en.html
##############################################################################
CONFIG proxy.config.cache.ram_cache.size INT -1
CONFIG proxy.config.cache.ram_cache_cutoff INT 4194304
    # https://docs.trafficserver.apache.org/records.config#proxy-config-cache-limits-http-max-alts
CONFIG proxy.config.cache.limits.http.max_alts INT 5
    # https://docs.trafficserver.apache.org/records.config#proxy-config-cache-max-doc-size
CONFIG proxy.config.cache.max_doc_size INT 0
CONFIG proxy.config.cache.min_average_object_size INT 8000
##############################################################################
# Logging Config. Docs:
#    https://docs.trafficserver.apache.org/records.config#logging-configuration
#    https://docs.trafficserver.apache.org/en/latest/reference/configuration/logs_xml.config.en.html
##############################################################################
CONFIG proxy.config.log.logging_enabled INT 3
CONFIG proxy.config.log.max_space_mb_for_logs INT 2000
CONFIG proxy.config.log.max_space_mb_headroom INT 1000
CONFIG proxy.config.log.squid_log_enabled INT 1
CONFIG proxy.config.log.squid_log_is_ascii INT 0
CONFIG proxy.config.log.rolling_enabled INT 1
CONFIG proxy.config.log.rolling_interval_sec INT 86400
CONFIG proxy.config.log.rolling_size_mb INT 10
CONFIG proxy.config.log.auto_delete_rolled_files INT 1
##############################################################################
# These settings control remapping, and if the proxy allows (open) forward proxy or not. Docs:
#    https://docs.trafficserver.apache.org/records.config#url-remap-rules
#    https://docs.trafficserver.apache.org/en/latest/reference/configuration/remap.config.en.html
##############################################################################
CONFIG proxy.config.url_remap.remap_required INT 0
    # https://docs.trafficserver.apache.org/records.config#proxy-config-url-remap-pristine-host-hdr
CONFIG proxy.config.url_remap.pristine_host_hdr INT 0
    # https://docs.trafficserver.apache.org/records.config#reverse-proxy
CONFIG proxy.config.reverse_proxy.enabled INT 0
##############################################################################
# SSL Termination. Docs:
#    https://docs.trafficserver.apache.org/records.config#client-related-configuration
#    https://docs.trafficserver.apache.org/en/latest/reference/configuration/ssl_multicert.config.en.html
##############################################################################
CONFIG proxy.config.ssl.client.verify.server INT 0
CONFIG proxy.config.ssl.client.CA.cert.filename STRING NULL
CONFIG proxy.config.ssl.server.cipher_suite STRING ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:AES128-GCM-SHA256:AES256-GCM-SHA384:ECDHE-RSA-RC4-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:RC4-SHA:RC4-MD5:AES128-SHA:AES256-SHA:DES-CBC3-SHA!SRP:!DSS:!PSK:!aNULL:!eNULL:!SSLv2
##############################################################################
# ICP Configuration. Docs:
#    https://docs.trafficserver.apache.org/records.config#icp-configuration
#    https://docs.trafficserver.apache.org/en/latest/reference/configuration/icp.config.en.html
##############################################################################
CONFIG proxy.config.icp.enabled INT 0
##############################################################################
# Debugging. Docs:
#    https://docs.trafficserver.apache.org/records.config#diagnostic-logging-configuration
##############################################################################
CONFIG proxy.config.diags.debug.enabled INT 0
CONFIG proxy.config.diags.debug.tags STRING http.*|dns.*
# ToDo: Undocumented
CONFIG proxy.config.dump_mem_info_frequency INT 0
CONFIG proxy.config.http.slow.log.threshold INT 0
##############################################################################
# Cluster Subsystem. Docs:
#    https://docs.trafficserver.apache.org/records.config#cluster
##############################################################################
LOCAL proxy.local.cluster.type INT 3
CONFIG proxy.config.cluster.ethernet_interface STRING lo
CONFIG proxy.config.cluster.cluster_port INT 8086
CONFIG proxy.config.cluster.rsport INT 8088
CONFIG proxy.config.cluster.mcport INT 8089
CONFIG proxy.config.cluster.mc_group_addr STRING 224.0.1.37
CONFIG proxy.config.body_factory.template_sets_dir STRING /etc/trafficserver/body_factory
CONFIG proxy.config.url_remap.filename STRING remap.config
CONFIG proxy.config.http_ui_enabled INT 1

CONFIG proxy.config.admin.user_id STRING trafficserver
CONFIG proxy.config.http.no_dns_just_forward_to_parent INT 1

A: I'd try a different curl:
curl --resolve example.com:8080:127.0.0.1 "https://exmaple.com/path"

I've never had success using the --proxy option for https requests.

A: I found that when using 127.0.0.1 or localhost as the proxy address I get
HTTP/1.1 502 Tunnel Connection Failed
but when using the fqdn or non-local ip address for the proxy things work much better.


text:
Q: 2 Elements next to image using Flex I'm trying to get the following result (using flexbox)

This is what I'm getting using my code below

I'm using the following code, which is aligning the image but it's applying the align-items to all the items so the other h6 is also floating..

.block {
  display: flex;
  align-items: center;
}

Jordan Baker

Owner of Bavet

Am I missing something? Thank you

A: You should put your headings in another wrapper

Jordan Baker

Owner of Bavet

A: With your provided markup, try the following

.block {
  display: flex;
  height: 100px;
  flex-direction: column;
  flex-wrap: wrap;
  justify-content: center;
  align-items: flex-start;
  width: 300px;
}

img {
  width: 100px;
  height: 100px;
}

h5, h6 {
  margin: 0;
  padding: 0;
}

Jordan Baker

Owner of Bavet

flex-direction and flex-wrap along with height of the block are the attributes that will help achieve the required alignment.
Change the margin, padding etc attributes of the h5, h6 or even the .block to your requirements.

redpajama_set_name: RedPajamaStackExchange


text:
Q: LDAP - Cannot Add OrganizationalUnit I am very new to LDAP and adding to my insanity.
I used the following code to create a database for a domain. Each domain will have its own database incase it is no longer valid etc.
version: 1

dn: olcDatabase={1}hdb,cn=config
objectClass: olcDatabaseConfig
objectClass: olcHdbConfig
olcDatabase: {1}hdb
olcDbDirectory: /data/ldap/example.intra
olcAccess: {0}to attrs=userPassword,shadowLastChange by self write by anonymous auth by dn="cn=admin,dc=example,dc=intra" write by * none
olcAccess: {1}to dn.base="" by * read
olcAccess: {2}to * by self write by dn="cn=admin,dc=example,dc=intra" write by * read
olcDbCheckpoint: 512 30
olcDbConfig: {0}set_cachesize 0 2097152 0
olcDbConfig: {1}set_lk_max_objects 1500
olcDbConfig: {2}set_lk_max_locks 1500
olcDbConfig: {3}set_lk_max_lockers 1500
olcDbIndex: objectClass                 eq
olcDbIndex: sn                          pres,sub,eq
olcDbIndex: uid                         pres,sub,eq
olcDbIndex: displayName                 pres,sub,eq
olcDbIndex: cn                          pres,sub,eq
olcDbIndex: default                     sub
olcDbIndex: uidNumber                   eq
olcDbIndex: gidNumber                   eq
olcDbIndex: mail,givenName          eq,subinitial
olcDbIndex: dc                      eq
olcLastMod: TRUE
olcRootDN: cn=admin,dc=example,dc=intra
olcRootPW: {SSHA}fJnMxL3dHhxKhDkFyK/MwKeRB4iY/NQ0
olcSuffix: dc=example,dc=intra

When I try and create an organizationalUnit with the following code it refuses:
version: 1

dn: olcDatabase={1}hdb, cn=admin, ou=People, dc=example,dc=intra
changetype: modify
add: objectClass
objectClass: organizationalUnit
ou: People
description: All People and Human Like Entities in the example.intra Organisation

Please could some really nice person show me what I need to write to get the update and if possible some content into this database?
I am using the following command line to do the work:
ldapadd -Y EXTERNAL -H ldapi:/// -f filename.ldif

This is the response that I get:
SASL/EXTERNAL authentication started
SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
SASL SSF: 0
ldapadd: wrong attributeType at line 16, entry "olcDatabase={1}hdb, cn=admin, ou=People, dc=example,dc=intra"

Thanks and kind regards,
jB


text:
Q: HAProxy - URL Based routing with load balancing - using listen and use-server I need to configure HAProxy to forward requests of different paths to different backends, and some of the backends need to be load-balanced. So I've come across this question, and the solution provided there does work, but due to our conventions, I need to use listen and use-server instead of frontend and use_backend.
So right now I have something like this:
listen poq [url]:[port]
    acl has_cool_url path_beg -i /cool
    use-server cool if has_cool_url
    server cool [ip]:[port] check
    server default [ip]:[port] check

And I cannot use this:
listen poq [url]:[port]
    acl has_cool_url path_beg -i /cool
    use_backend cool if has_cool_url
    use_backend notcool if !has_cool_url

backend cool
    balance roundrobin
    server first [ip]:[port] check
    server second [ip]:[port] check

backend notcool
    server third [ip]:[port] check

Because our conventions tell us to define everything within the listen block.
So my question is: is there anything like this:
listen poq [url]:[port]
    acl has_cool_url path_beg -i /cool
    use-server {first, second} if has_cool_url
    server first [ip]:[port] check
    server second [ip]:[port] check
    server default [ip]:[port] check

Where first and second are load-balanced using round robin?

A: Your conventions are arbitrary and are limiting your functionality.  A listen block is functionally similar to a single frontend and a backend block with the same name, and an implicit default_backend pointing from frontend to backend.  listen blocks are intended to be used for simple configurations.
The point of a backend is that all servers in the back-end provide the same service.  use-server allows you to select exactly one server.
The only thing you can do with listen is balance a single default set of servers, plus one or more individual servers, individually addressable using condition matches, but they can't be balanced.
listen poq [url]:[port]
    acl has_warm_url path_beg -i /warm
    use-server warm-server if has_warm_url
    acl has_hot_url path_beg -i /hot
    use-server hot-server if has_hot_url
    server warm-server [ip]:[port] check weight 0
    server hot-server [ip]:[port] check weight 0
    server default1 [ip]:[port] check
    server default2 [ip]:[port] check

In this configuration, /warm goes to warm-server, /hot goes to hot-server.  These two servers are weighted 0 to prevent them from handling any other requests.
Everything else is balanced between default1 and default2.
This is all the flexibility you have with listen because HAProxy only does one set of balancing per backend, and, as noted above, a listen section is an implicit combination of the functionality of a matched set of frontend and backend.


text:
Q: Definition of homogeneous or quasi-uniform or almost uniform measure Let us call a measure $\Lambda$ homogeneous if there is an $\epsilon>0$ so that for all $r>0$ and $x,y$ in the support of $\Lambda$, we have
$$\Lambda(B(x,r))>\epsilon\Lambda(B(y,r))$$
where as usual $B(x,r)$ is the ball of radius $r$ centred at $x$.  I am thinking of $\mathbb{R}^d$ but it could be a general metric space.  Q: Is this (equivalent to) a known definition in the literature?
Update: I followed up dirk's comment to user116082's answer.  Google scholar gives me exactly 10 results for "quasi uniform measure."  The two that seem relevant are
Saul Jacka and Jon Warren, "Random orderings of the integers and card shuffling," Stochastic Processes and their Applications, 117, 708-719 (2007).
which is quite a different concept.  Also
É Janvresse and T. de la Rue, "A class of pairwise-independent joinings," Ergodic Theory and Dynamical Systems, 28, 1545-1557 (2008).
which is very close to what I have applied to cylinder sets in symbolic dynamics.  The latter paper gives the definition but is worded in a way that suggests it is a known concept.  The relevant reference [5] in that paper does not appear to contain this terminology, though (searching using google books).
So, for now, the original question still stands, for more general contexts than symbolic dynamics, and also Q2: What is the first usage of the term quasi-uniform measure as defined in the second paper above?
Update 2: Repeating the above process for "almost uniform measure" yields 11 google scholar results, one of which gives a definition that looks equivalent to the above, except for all points in the space, not just in the support of the measure.
For example, the usual measure on the middle third Cantor set satisfies the above definition in $\mathbb{R}$ but the "almost uniform measure" definition only if the space is the Cantor set itself with metric induced from its embedding into $\mathbb{R}$.
The relevant reference is
M. Studený, "On the differentiation theorem in metric groups." Commentationes Mathematicae Universitatis Carolinae 24 223-232 (1983).
On balance, it seems best to stick with "almost uniform measure" but note the above distinction with regard to the support.  I am still interested in any further references or information about these definitions.

A: I can't comment, being anonymous, and this is not a true answer to the question.
Your notion is similar to the notion of a uniformly distributed measure used in geometric measure theory. That is a measure that assigns the (exact) same mass to all balls of the same radius.
Your measure only assigns uniformly comparable mass to all balls of the same radius. I might therefore call it ``quasi-uniformly distributed'' although this is a bit of a mouthful.


text:
Q: Error in FD package in R: Species labels in 'x' and 'a' need to be identical and ordered alphabetically (or simply in the same order) I saw that others had this problem before but the solutions they found didn't work for me so far unfortunately.
I am trying to find my way around the FD package and to reproduce some examples it gives with my own data. I read "How to make a great R reproducible example" and hope to do it right.
Here is a short version of my data:
dput(wood_trait[1:4,1:4])
structure(list(family = c("Sapindaceae", "Sapindaceae", "Sapindaceae",
"Sapindaceae"), growth = c("T", "T", "T", "T"), pollsyn = c("entomophilous",
"both", "entomophilous", "entomophilous"), beepoll = c(" 1",
" 1", NA, " 1")), row.names = c("Acercampestre", "Acernegundo",
"Acerpalmatum", "Acerplatanoides"), class = "data.frame")

dput(wood_abun[,1:4])
structure(c(" 0", " 0", " 0", "11", " 0", " 0", " 1", " 0", " 0",
" 0", " 0", " 0", " 0", " 0", " 2", " 0", " 3", " 0", " 1", " 0",
" 0", " 1", "13", " 0", " 1", " 1", "10", " 0", " 0", " 1", " 6",
" 0", " 1", " 0", " 0", " 2", "0", "0", "0", "0", "0", "0", "0",
"3", "0", "0", "0", "0", "0", "0", "0", "0", "0", "0", " 0",
" 0", " 0", " 0", " 4", " 6", " 0", " 0", "53", " 0", " 4", " 6",
" 0", " 2", " 0", " 1", " 0", " 2"), .Dim = c(18L, 4L), .Dimnames = list(
    c("AllmendeKontor", "GASFK", "GASTS", "GartenderBegegnung",
    "Himmelbeet", "Inselgarten", "Fischerstrasse", "Klunker",
    "PeaceofLand", "VatNeuGruenstr", "Prinzessin", "RoteBeete",
    "Schalotten", "BurbacherWeg", "SpielFeld", "Wachsenlassen",
    "Vollguter", "Wieckerstrasse"), c("Acercampestre", "Acernegundo",
    "Acerpalmatum", "Acerplatanoides")))

When running:
exwood2 <- functcomp(wood_trait, wood_abun)

I get the following error message:
Species labels in 'x' and 'a' need to be identical and ordered alphabetically (or simply in the same order).
I have checked multiple times with excel that the species labels are identical. I am a bit afraid that the problem lies somewhere else in the data than in the subset I am posting here.. But I am not sure how to find it. Also, doesn't this
setdiff(attr(wood_trait, 'Labels'), colnames(wood_abun))

NULL
suggest that there is no difference between the rownames and column names? But:
identical(row.names(wood_trait), colnames(wood_abun))

[1] FALSE
I would very much appreciate some help!

A: I finally figured it out! I created subsets of only a few rows/columns at a time and compared them using identical() as described above. I finally managed to find where the error was, found it in excel and changed it and now it works.
The data was indeed the same in both excel sheets, but it had a - sign in it and taking it out solved the problem.


text:
Q: Airy functions identities Playing around with the Airy equation $$y''=t y$$
I came across determinant expressions of the form
$$\text{Ai}(x) \text{Bi}(y)-\text{Ai}(y) \text{Bi}(x), $$
as well as
$$ \text{Ai}(x) \text{Bi}'(y)-\text{Ai}'(y) \text{Bi}(x).$$
I was wondering if there exist any "Airy identities", similarly to the trigonometric case, which could help me simplify such expressions.
Thank you!


text:
Q: Problem reading JPEG Metadata (Orientation) I've got a JPEG image which was taken on an iphone. On my desktop PC (Windows Photo Viewer, Google Chrome, etc) the orientation is incorrect.
I'm working on an ASP.NET MVC 3 web application where i need to upload photos (currently using plupload).
I've got some server-side code to process images, including reading EXIF data.
I've tried reading the PropertyTagOrientation field in the EXIF meta data (using GDI - Image.PropertyItems), but the field isn't present.
So it's either some specific iphone meta data, or some other meta data.
I've used another tool like Aurigma Photo Uploader, and it reads the meta data correctly and rotates the image. How does it do this?
Does anyone know what other JPEG meta data could contain the information required in order to know that it needs to be rotated, that is used by Aurigma?
Here's the code i'm using to read the EXIF data:
var image = Image.FromStream(fileStream);

foreach (var prop in image.PropertyItems)
{
   if (prop.Id == 112 || prop.Id == 5029)
   {
      // do my rotate code - e.g "RotateFlip"
      // Never get's in here - can't find these properties.
   }
}

Any ideas?

A: Posting here just in case someone has the same issue. I had problems in production reading the orientation using WFP and GDI. The only thing that worked was using: https://github.com/dlemstra/Magick.NET
The code is fairly simple:
var img = new MagickImage(inputStream);
img.AutoOrient();   // Fix orientation
img.Strip();        // remove all EXIF information
img.Write(outputPath);

A: It appears that you forgotten that the orientation id values you looked up are in hex. Where you use 112, you should have used 0x112.
This article explains how Windows ballsed-up orientation handing, and this one seems pretty relevant to what you are doing.

A: Here is a snippet addressing the 8 orientation values.
First a few notes:
The EXIF id 0x0112 is for Orientation. This is a helpful EXIF id reference http://www.sno.phy.queensu.ca/~phil/exiftool/TagNames/EXIF.html
0x0112 is the hex equivalent of 274. The data type of a PropertyItem.Id is an int, meaning 274 is what is useful here.
Additionally, 5029 likely was supposed to be 0x5029 or 20521 which correlates to ThumbnailOrientation, though is likely not what is desired here.
Orient Image:
Note: img is a System.Drawing.Image or inherits from it, like System.Drawing.Bitmap.

if (Array.IndexOf(img.PropertyIdList, 274) > -1)
{
    var orientation = (int)img.GetPropertyItem(274).Value[0];
    switch (orientation)
    {
        case 1:
            // No rotation required.
            break;
        case 2:
            img.RotateFlip(RotateFlipType.RotateNoneFlipX);
            break;
        case 3:
            img.RotateFlip(RotateFlipType.Rotate180FlipNone);
            break;
        case 4:
            img.RotateFlip(RotateFlipType.Rotate180FlipX);
            break;
        case 5:
            img.RotateFlip(RotateFlipType.Rotate90FlipX);
            break;
        case 6:
            img.RotateFlip(RotateFlipType.Rotate90FlipNone);
            break;
        case 7:
            img.RotateFlip(RotateFlipType.Rotate270FlipX);
            break;
        case 8:
            img.RotateFlip(RotateFlipType.Rotate270FlipNone);
            break;
    }
    // This EXIF data is now invalid and should be removed.
    img.RemovePropertyItem(274);
}

A: From this post looks like you need to check ID 274
foreach (PropertyItem p in properties) {
      if (p.Id == 274) {
            Orientation = (int)p.Value[0];
         if (Orientation == 6)
            oldImage.RotateFlip(RotateFlipType.Rotate90FlipNone);
         if (Orientation == 8)
            oldImage.RotateFlip(RotateFlipType.Rotate270FlipNone);
      break;
      }
}

A: I combined the given answers and comments and came up this:
    MemoryStream stream = new MemoryStream(data);
    Image image = Image.FromStream(stream);

    foreach (var prop in image.PropertyItems) {
        if ((prop.Id == 0x0112 || prop.Id == 5029 || prop.Id == 274)) {
            var value = (int)prop.Value[0];
            if (value == 6) {
                image.RotateFlip(RotateFlipType.Rotate90FlipNone);
                break;
            } else if (value == 8) {
                image.RotateFlip(RotateFlipType.Rotate270FlipNone);
                break;
            } else if (value == 3) {
                image.RotateFlip(RotateFlipType.Rotate180FlipNone);
                break;
            }
        }
    }


text:
Q: How to return a promise from a Firebase Firestore set() call? This is a followup to my question https://stackoverflow.com/questions/54100270/google-translate-api-and-firebase-firestore-are-killing-each-other. Doug Stevenson said that I needed to return my functions. We watched his video Learn JavaScript Promises (Pt. 3) for sequential and parallel work in Cloud Functions a few hundred times...
I'm working with two functions. The first, a call to Google Translate, now returns a translation. The second function is a Firestore set() call, to write the translation to the database. The set() works if I don't do the Google Translate call, but together they crash. Specifically, if I call Firebase set() than the Google Translate function executes, we see Result1, and nothing further. In other words, calling the database stops to code from pushing a translation into the translationArray.
Here's my code now:
exports.Google_EStranslateEN = functions.firestore.document('Users/{userID}/Spanish/Translation_Request_NOT').onUpdate((change, context) => {
    if (change.after.data().word != undefined) {
        const { Translate } = require('@google-cloud/translate');

        // Your Google Cloud Platform project ID
        const projectId = 'myProject-cd99d';

        // Instantiates a client
        const translate = new Translate({
            projectId: projectId,
        });

        // The text to translate
        const text = change.after.data().word;
        console.log(text);
        // The target language
        const target = 'en';

        let translationArray = []; // clear translation array

        const finalPromise = translate.translate(text, target)
            .then(function (results) {
                console.log("Result1: " + results);
                console.log(Array.isArray(results));
                console.log(results[0]);
                let translation = results[0];
                console.log(translation);
                return translation
            })
            .then(function (results) {
                console.log("Translation: " + results);
                translationArray.push(results);
                return translationArray
            })
            .then(function (results) {
                console.log("TranslationArray: " + translationArray);
                console.log("Result2: " + results);
                console.log("Text: " + text)
                return admin.firestore().collection('Dictionaries').doc('Spanish').collection('Words').doc(text).collection('Translations').doc('English').set({
                    translationArray: results,
                    language: 'en',
                    longLanguage: 'English'
                });
            })
            .then(function (results) {
                console.log("Result3: " + results)
                console.log("Write succeeded!");
            })
            .catch(function (error) {
                console.error(error);
            });
    } // close if
    return 0;
}); // close oxfordPronunciationUS

The Firestore set() call returns nothing, and it kills the Google Translation call. Specifically, without the Firestore call the code executes all the way through, logging everything. With the Firestore call, neither function executes (Google Translate is never called) and nothing logs after "text."
I don't understand what const finalPromise does. It looks like an unused constant.
We read Why are the Firebase API asynchronous? and tried this code:
var promise = admin.firestore().collection('Dictionaries').doc('Spanish').collection('Words').doc(text).collection('Translations').doc('English').set({
    translationArray: translationArray,
    language: 'en',
    longLanguage: 'English'
});
promise.then(...)

That didn't help.
I tried switching to IBM Watson Translate, and the same thing is happening. Without the database call the translation function works perfectly. With the Firestore call it gets the translation, runs the forEach, and then stops when I try to push the word into the array. translationsArray doesn't log and nothing writes to the database.
exports.IBM_EStranslateEN = functions.firestore.document('Users/{userID}/Spanish/Translation_Request_IBM').onUpdate((change, context) => {
    if (change.after.data().word != undefined) {

        let word = change.after.data().word;
        let wordArray = [];
        wordArray.push(word);

        var LanguageTranslatorV3 = require('watson-developer-cloud/language-translator/v3');

        var languageTranslator = new LanguageTranslatorV3({
            version: '2018-05-01',
            iam_apikey: 'swordfish',
            url: 'https://gateway.watsonplatform.net/language-translator/api',
            headers: {
                'Content-Type': 'application/json'
            }
        });

        var parameters = {
            "text": wordArray,
            "model_id": "es-en",
            "source": "es",
            "target": "en"
        };

        let translationsArray = [];

        languageTranslator.translate(
            parameters,
            function (err, response) {
                if (err) {
                    console.log('error:', err);
                } else {
                    response.translations.forEach(function (translation) {
                        console.log(translation.translation);
                        translationsArray.push(translation.translation);
                    });
                    console.log(translationsArray);

                    admin.firestore().collection('Dictionaries').doc('Spanish').collection('Words').doc(word).collection('Translations').doc('English').set({
                        translationsArray: translationsArray,
                        language: 'en',
                        longLanguage: 'English'
                    })
                        .then(function () {
                            console.log("Translations written to database.");
                        })
                        .catch(function (error) {
                            console.error(error);
                        });

                }
            }
        );
    }
    return 0;
});

I also wrote the same cloud function calling the Oxford English Dictionary for the translation. This works perfectly, writing the translation to the database:
exports.Oxford_EStranslateEN = functions.firestore.document('Users/{userID}/Spanish/Translation_Request').onUpdate((change, context) => {
    if (change.after.data().word != undefined) {
        let word = change.after.data().word;
        let options = {
            uri: 'https://od-api.oxforddictionaries.com/api/v1/entries/es/' + change.after.data().word + '/translations%3Den', // translations=es
            headers: {
                "Accept": "application/json",
                'app_id': 'groucho',
                'app_key': 'swordfish'
            },
            json: true
        };
        let translationsArray = [];
        return rp(options)
            .then(function (wordObject) {
                wordObject.results.forEach(function (result) {
                    result.lexicalEntries.forEach(function (lexicalEntry) {
                        lexicalEntry.entries.forEach(function (entry) {
                            entry.senses.forEach(function (sense) {
                                if (sense.translations) {
                                    sense.translations.forEach(function (translation) {
                                        translationsArray.push(translation.text);
                                    });
                                } // close if
                                else {
                                    if (sense.subsenses) {
                                        sense.subsenses.forEach(function (subsense) {
                                            if (subsense.translations) {
                                                subsense.translations.forEach(function (translation) {
                                                    translationsArray.push(translation.text);
                                                }); // close forEach
                                            } // close if
                                            else {
                                                // console.log("No Translation");
                                            } // close else
                                        }); // close forEach
                                    } // close if
                                } // close else
                            }); // close forEach
                        }); // close forEach
                    }); // close forEach
                }); // close forEach
                translationsArray = [...new Set(translationsArray)]; // removes duplicates
                return admin.firestore().collection('Dictionaries').doc('Spanish').collection('Words').doc(word).collection('Translations').doc('English').set({
                    translationsArray: translationsArray,
                    source: 'OED',
                    dateAdded: Date.now(), // timestamp
                    longLanguage: 'English',
                    shortLanguage: 'en',
                    word: word
                })
                    .then(function () {
                        // console.log("Document written.");
                    })
                    .catch(function (error) {
                        console.log("Error writing document: ", error);
                    })
            })
            .then(function () {
                // console.log("Document written for Oxford_EStranslateEN.");
            })
            .catch(function (error) {
                console.log("error: " + error);
            });
    } // close if
    // return 0;
});

One difference is that I call the OED via an HTTP request, using rp (request-promise). I call return rp(options). This clearly returns a promise, and the promise is explicitly returned. The problem seems to be that in the Google version I'm not returning the promise when I call Google Translate, and IBM Watson returns a callback, not a promise, and I don't return that.

A: The Firebase JavaScript (Web) SDK .set method returns a promise, i.e., the promise is already there. See Set A Document online documentation.

// Add a new document in collection "cities"
db.collection("cities").doc("LA").set({
    name: "Los Angeles",
    state: "CA",
    country: "USA"
})
.then(function() {
    console.log("Document successfully written!");
})
.catch(function(error) {
    console.error("Error writing document: ", error);
});

You might find async/await easier to use (more intuitive).  In that scenario, you would do something like:
async function writeDocument(obj){
  var writeOperation = await db.collection("cities").doc("LA").set(obj);
  //now this code is reached after that async write
}

A: I got it working:
exports.Google_EStranslateEN = functions.firestore.document('Users/{userID}/Spanish/Translation_Request_Google').onUpdate((change, context) => {
  if (change.after.data().word != undefined) {
    // copied from https://cloud.google.com/translate/docs/quickstart-client-libraries
    // Imports the Google Cloud client library
    const {Translate} = require('@google-cloud/translate');
    // Your Google Cloud Platform project ID
    const projectId = 'myProject-cd99d';
    // Instantiates a client
    const translate = new Translate({
      projectId: projectId,
    });
    const word = change.after.data().word; // The word to translate
    const options = {
      from: 'es', // the source language
      to: 'en', // the target language
      format: 'text' // HTML vs. plain text
    };
    let translationArray = [];  // clear translation array
    return translate.translate(word, options)  // this return is critical
    .then(function(results) {
      let translation = results[0];
      translationArray.push(translation);
      admin.firestore().collection('Dictionaries').doc('Spanish').collection('Words').doc(word).collection('Translations').doc('English').set({
        translationArray: translationArray,
        language: 'en',
        longLanguage: 'English'
      });
    })
    .then(function() {
      console.log("Write succeeded!");
    })
    .catch(function(error) {
      console.error(error);
    });
  } // close if
});

The key was to return the translate function:
    return translate.translate(word, options)

The cloud function also runs faster.
Doug Stevenson answered my previous question:

You're not returning a promise that's resolved when all the async work
  is complete. If you don't do that, Cloud Functions assumes that all
  your work is complete, and will clamp down on all resources, and any
  pending work will be shut down.
The promise returned by translate.translate().then().catch() is being
  ignored. Your nested call to admin.firestore()...set() has a similar
  problem. It's not sufficient to just call then() and catch() on every
  promise because then() and catch() both return yet another promise.

It seems like the phrase "return a promise" is being used in two ways here. Saying that translate.translate() returns a promise is different from saying that I need to return a promise by coding return translate.translate(). Perhaps it would have been clearer if Doug had said to "return the function" instead of "return a promise"?
I'm also not clear on whether I need to return this function:
return admin.firestore()...set({

It works the same with or without the return.


text:
Q: Using variable to retrieve array element inside awk statement Below is the part of the code that is throwing error.
for (( p=0; p<${var2}; p++ ))
do

if [[ ${array2[$p]} == *"xyz"* ]]
then
    awk -F" " '{ array3[p]="123" };1' ${array1[$i]}
else
    awk -F" " '{ array3[p]="456" };1' ${array1[$i]}
fi

When i run the script in debug mode, i could see that the value of p is not being considered.
+ awk '-F ' '{ array3[p]="456" };1' oct.csv

I am running the script as ./script.sh and /bin/ksh is the shell used. Please help.

A: First of all, this is wrong:
awk -F" " '{ array3[p]="123" };1' ${array1[$i]}

${array1[$i]} is not a file, so you can't just throw it as first command line argument to awk. With variables you'd have to use either echo $VARIABLE or here string <<< like so
awk -F" " '{ array3[p]="123" };1' <<< ${array1[$i]}

Second, of all, there is a scope issue. array3[p]="123" only exists within awk code, so you can't assign anything to an outside of it. What you could do, however, simply do
array3[$p]="123"

It is also unclear where exactly $i comes from in your script. Overall, your question gives us a bit of code but doesn't provide any clues to fully answer your question


text:
Q: Using scopes to filter results in rails 5 I'm running into an issue with scopes in rails.  Currently I have a case statement in the index controller which is causing the app to either filter on 1 category OR another category.  But I need to be able to filter on multiple things at once.  Basically to chain the scopes so when a user clicks on link A, scope A is applied and then when they click on link B scope B is added on top of scope A to filter even more.  Eventually I would like to add a check box in the view so when it's checked the scope is applied and when unchecked the scope is removed.
I have a bunch of conditional logic (if\else) statements that I use now because I don't just need to filter on say "experience" I need to filter on 4 different options of experience (0-2 years, 2-5 years, 5-10 years, etc...).  This will be the same for all scopes.
So in summary I need to find a way to chain scopes so that the filters keep getting applied (filter A + filter B + filter C, etc...) and not have them the way they are now where it's either filter A or filter B but can't get them both to work together to further narrow down results.
here are my scopes
scope :by_experience, -> (ex) { where(experience: ex) }

scope :by_num_days_past, -> (days_past) { where('created_at >= ?', days_past.days.ago) }

here is my view

                Experience

                <%= link_to "0-2 years", filtered_jobs_path(:experience, '0-2 years') %>

                <%= link_to "2-5 years", filtered_jobs_path(:experience, '2-5 years') %>

                <%= link_to "5-10 years", filtered_jobs_path(:experience, '5-10 years') %>

                <%= link_to "10+ years", filtered_jobs_path(:experience, '10+ years') %>

                Days Listed

                <%= link_to "7 days", filtered_jobs_path(:num_days_past, '7') %>

                <%= link_to "30 days", filtered_jobs_path(:num_days_past, '30') %>

                <%= link_to "60 days", filtered_jobs_path(:num_days_past, '60') %>

Here's my controller's index action where I think the change needs to take place in order for there to be multiple scopes applied at once.
def index

      case params[:scope]
      when "num_days_past"
        if (params[:format] == "7")
          @jobs = Job.by_num_days_past("7").paginate(page: params[:page], per_page: 5)
        elsif (params[:format] == "30")
          @jobs = Job.by_num_days_past("30").paginate(page: params[:page], per_page: 5)
        elsif (params[:format] == "60")
          @jobs = Job.by_num_days_past("60").paginate(page: params[:page], per_page: 5)
         else
             @jobs = Job.order('created_at DESC').paginate(page: params[:page], per_page: 5)
         end
end

A: Calling multiple scopes on the instance variable gets them chained. Please try following:
def index

    @jobs = Job.by_num_days_past("7") if (params[:format] == "7")
    @jobs = @jobs&.by_num_days_past("30") || Job.by_num_days_past("30") if (params[:format] == "30")
    @jobs = @jobs&.by_num_days_past("60") || Job.by_num_days_past("60") if (params[:format] == "60")
    @jobs = Job.order('created_at DESC') if @jobs.blank?
    @jobs.paginate(page: params[:page], per_page: 5)

end

A: You could delay the pagination till you're done applying scopes. Like so:
def index
  case params[:scope]
  when "num_days_past"
    if (params[:format] == "7")
      @jobs = Job.by_num_days_past("7")
    elsif (params[:format] == "30")
      @jobs = Job.by_num_days_past("30")
    elsif (params[:format] == "60")
      @jobs = Job.by_num_days_past("60")
    else
      @jobs = Job.all
    end

  # Now you can apply scope 2
  # For example
  @jobs = @jobs.by_experience(params[:exp]) if params[:scope_by_experience]

  @jobs = @jobs.order('created_at DESC').paginate(page: params[:page], per_page: 5)
end

The #paginate implementation might be executing the query instantly and the result wouldn't be an Active Record object that you can apply further scopes on.

A: So I ended up using a combination of several of the answers.  Thanks to Oscar for explaining about holding off pagination until last, Ahmand thanks for the info about chaining, Johan thanks for explaining about the jobs_path params.  I would up vote but it says I don't have enough rep points yet.  Posting my answer here in case anyone stumbles upon this in the future.
In the controller I start with Job.all and then build the scopes on top of that, finally applying the pagination.  I feel like I should move all this to a method in the model though and then just call it from the controller.  But it's working the way it is, I'll test moving it to the model later.
def index
  @jobs = Job.all

  # scopes
  if params[:experience].present?
    @jobs = @jobs.by_experience(params[:experience])
  end

  if params[:num_days_past].present?
    @jobs = @jobs.by_num_days_past(params[:num_days_past].to_i)
  end

  @jobs = @jobs.paginate(page: params[:page], per_page: 5)
end

In my view I have added the experience params to the link so that the url will display both experience and filter by days if the parameter is present.

            <%= link_to "7 days", filtered_jobs_path(num_days_past: '7', experience: params[:experience]) %>

            <%= link_to "30 days", filtered_jobs_path(num_days_past: '30', experience: params[:experience]) %>

            <%= link_to "60 days", filtered_jobs_path(num_days_past: '60', experience: params[:experience]) %>

Here's my model with the scopes
 # scopes to filter results from database.
scope :by_created_at, -> { order(created_at: :desc) }
scope :by_experience, -> (ex) { where(experience: ex) if ex.present? }
scope :by_num_days_past, -> (days_past) { where('created_at >= ?', days_past.days.ago) if days_past.present? }
scope :by_location, -> (location) { where('city LIKE ? OR state LIKE ?', "%#{location}%", "%#{location}%") }

Here are my routes, I removed the /filer/:scope that I previously had.
  #get 'jobs/filter/:scope' => 'jobs#index', as: :filtered_jobs
  get 'jobs' => 'jobs#index', as: :filtered_jobs

A:
The problem:

The reason it currently doesn't work is you don't save the current filter/scope. If you click a second filter you have no clue what scope was previously rendered.
To fix this issue you need to store the filter on the client and send it as param on the next request. This is generally done in the URL, this way you can share the website with filters. If someone else goes to the link the filters automatically jump to the correct position. But it can also be done through cookies.
For example take a look at:

*

*https://tweakers.net/categorie/496/laptops/producten/

*https://www.laptopshop.nl/category/45702/laptops.html (less cryptic)

I know they may be foreign languages to you (Dutch), but take a close look at the URL bar when turning filters on and off. The filters are saved and passed on with the next request you make.

A posible solution:

Currently you created a dedicated route for the filter, my guess is:
resources :jobs do
  get '/:scope/:format', on: :collection, as: :filtered, action: :index
end

What you should do is use the jobs_path with params like so (view):
jobs_path(number_of_days_past: 7, experience: '0-2 years')

This will generate the URL:
/jobs?number_of_days_past=7&experience=0-2+years

If you want to combine a new filter with previous filters you have to extract the previous filters first. This is done in the controller like so:
def index
  @filters = params.permit(:number_of_days_past, :experience).to_h
end

Create a helper method for combining the @filters with the filters you want to apply or remove (app/helpers/jobs_helper.rb). The example below can toggle a filter on and off and supports multiple filters, but not multiple values for a given filter. If you want support for multiple values per filter, take this as an example and edit it to your liking.
def combine_filter(apply_filters)
  apply_filters.each_with_object(@filters.deep_dup) do |(key, value), filters|
    filters[key] == value ? filters.delete(key) : filters[key] = value
  end
end

# OR, if the above did go to fast.

def combine_filter(apply_filters)
  # duplicate the @filters hash, this way multiple calls don't influence each other
  filters = @filters.deep_dup
  # loop through apply_filters (Hash) and check every filter
  apply_filters.each do |key, value|
    if filters[key] == value
      # filter content was the same, toggle it off
      filters.delete(key)
    else
      # filter content was not the same, replace value for this filter
      filters[key] = value
    end
  end
  # return the filters variable
  filters
end

This will leave you with a view like this:

    <%= link_to "0-2 years", jobs_path(combine_filter(experience: '0-2 years')) %>

    <%= link_to "2-5 years", jobs_path(combine_filter(experience: '2-5 years')) %>

    <%= link_to "5-10 years", jobs_path(combine_filter(experience: '5-10 years')) %>

    <%= link_to "10+ years", jobs_path(combine_filter(experience: '10+ years')) %>

At this point when receiving a request in the controller you have all filters in the variable @filters but still need to handle them. Check the online resources below for an example.

Online help and resources:

Here is a blogpost on the how to implement a simple version (handling the incoming filters and calling the correct scopes):

*

*https://www.justinweiss.com/articles/search-and-filter-rails-models-without-bloating-your-controller/
But you could also try to find the right gem for the job, here is one that I found with a quick Google search:

*

*https://github.com/jhund/filterrific


text:
Q: Python Threading and interpreter shutdown- Is this fixable or issue Python Issue #Issue14623 I have python script that uploads files to a cloud account. It was working for a while, but out of now where I started getting the 'Exception in thread Thread-1 (most likely raised during interpreter shutdown)' error. After researching I found this python issue http://bugs.python.org/issue14623 which states the issue will not get fixed.
However, I'm not exactly sure this would apply to me and I am hoping someone could point out a fix. I would like to stay with python's threading and try to avoid using multiprocessing since this is I/O bound. This is the stripped down version(which has this issue also), but in the full version the upload.py has a list I like to share so I want it to  run in the same memory.
It always breaks only after it completes and all the files are uploaded. I tried removing 't.daemon = True' and it will just hang(instead of breaking) at that same point(after all the files are uploaded). I also tried removing q.join() along with 't.daemon = True' and it will just hang after completion. Without the t.daemon = True and q.join(), I think it is blocking at item = q.get() when it comes to the end of the script execution(just guess).
main:
 import logging
import os
import sys
import json
from os.path import expanduser
from Queue import Queue
from threading import Thread
from auth import Authenticate
from getinfo import get_containers, get_files, get_link
from upload import upload_file
from container_util import create_containers
from filter import MyFilter

home = expanduser("~") + '/'

directory = home + "krunchuploader_logs"

if not os.path.exists(directory):
    os.makedirs(directory)

debug = directory + "/krunchuploader__debug_" + str(os.getpid())
error = directory + "/krunchuploader__error_" + str(os.getpid())
info = directory + "/krunchuploader__info_" + str(os.getpid())

os.open(debug, os.O_CREAT | os.O_EXCL)
os.open(error, os.O_CREAT | os.O_EXCL)
os.open(info, os.O_CREAT | os.O_EXCL)

formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')

logging.basicConfig(level=logging.DEBUG,
    format='%(asctime)s - %(levelname)s - %(message)s',
    filename=debug,
    filemode='w')

logger = logging.getLogger("krunch")

fh_error = logging.FileHandler(error)
fh_error.setLevel(logging.ERROR)
fh_error.setFormatter(formatter)
fh_error.addFilter(MyFilter(logging.ERROR))

fh_info = logging.FileHandler(info)
fh_info.setLevel(logging.INFO)
fh_info.setFormatter(formatter)
fh_info.addFilter(MyFilter(logging.INFO))

std_out_error = logging.StreamHandler()
std_out_error.setLevel(logging.ERROR)
std_out_info = logging.StreamHandler()
std_out_info.setLevel(logging.INFO)

logger.addHandler(fh_error)
logger.addHandler(fh_info)
logger.addHandler(std_out_error)
logger.addHandler(std_out_info)

def main():
    sys.stdout.write("\x1b[2J\x1b[H")
    print title
    authenticate = Authenticate()
    cloud_url = get_link(authenticate.jsonresp)
    #per 1 million files the list will take
    #approx 300MB of memory.
    file_container_list, file_list = get_files(authenticate, cloud_url)
    cloud_container_list = get_containers(authenticate, cloud_url)
    create_containers(cloud_container_list,
        file_container_list, authenticate, cloud_url)

    return file_list

def do_the_uploads(file_list):
    def worker():
    while True:
        item = q.get()
        upload_file(item)
        q.task_done()

    q = Queue()
    for i in range(5):
    t = Thread(target=worker)
    t.daemon = True
    t.start()

    for item in file_list:
    q.put(item)

    q.join()

if __name__ == '__main__':
    file_list = main()
    value = raw_input("\nProceed to upload files? Enter [Y/y] for yes: ").upper()
    if value == "Y":
    do_the_uploads(file_list)

upload.py:
def upload_file(file_obj):
    absolute_path_filename, filename, dir_name, token, url = file_obj
    url = url + dir_name + '/' + filename

    header_collection = {
        "X-Auth-Token": token}

    print "Uploading " + absolute_path_filename
    with open(absolute_path_filename) as f:
    r = requests.put(url, data=f, headers=header_collection)
    print "done"

Error output:
Fetching Cloud Container List... Got it!
All containers exist, none need to be added

Proceed to upload files? Enter [Y/y] for yes: y
Uploading /home/one/huh/one/green
Uploading /home/one/huh/one/red
Uploading /home/one/huh/two/white
Uploading /home/one/huh/one/blue
 Uploading /home/one/huh/two/yellow
done
Uploading /home/one/huh/two/purple
done
done
done
done
done
Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 808, in __bootstrap_inner
  File "/usr/lib64/python2.7/threading.py", line 761, in run
  File "krunchuploader.py", line 97, in worker
  File "/usr/lib64/python2.7/Queue.py", line 168, in get
  File "/usr/lib64/python2.7/threading.py", line 332, in wait
<type 'exceptions.TypeError'>: 'NoneType' object is not callable

UPDATE: I placed a time.sleep(2) at the end of the script which seems to have fixed the issue. I guess the sleep allows the daemons to finish before the script comes to end of life and closes? I would of thought the main process would have to wait for the daemons to finish.

A: You can use a "poison pill" to kill the workers gracefully. After putting all the work in the queue, add a special object, one per worker, that workers recognize and quit. You can make the threads non-daemonic so Python will wait for them to finish before shutting down the process.
A concise way to make worker recognize poison and quit is to use the two-argument form of the iter() builtin in a for loop:
def do_the_uploads(file_list):
    def worker():
        for item in iter(q.get, poison):
            upload_file(item)

    poison = object()
    num_workers = 5
    q = Queue()
    for i in range(num_workers):
        t = Thread(target=worker)
        t.start()

    for item in file_list:
        q.put(item)

    for i in range(num_workers):
        q.put(poison)


text:
Q: How to save a spark rdd to an avro file I am trying to save an rdd to a file in avro format. This is how my code looks like:
val output = s"/test/avro/${date.toString(dayFormat)}"
  rmr(output)//deleteing the path
  rdd.coalesce(64).saveAsNewAPIHadoopFile(
    output,
    classOf[org.apache.hadoop.io.NullWritable],
    classOf[PageViewEvent],
  classOf[AvroKeyValueOutputFormat[org.apache.hadoop.io.NullWritable,PageViewEvent]],
    spark.hadoopConfiguration)
}

When I run this I get an error saying:
 Unsupported input type PageViewEvent

The type of the rdd is RDD[(Null,PageViewEvent)].
Can someone explain my what I am doing wrong?
Thanks in advance

A: So I managed to find a 'workaround'.
 val job = new Job(spark.hadoopConfiguration)
  AvroJob.setOutputKeySchema(job, PageViewEvent.SCHEMA$)

  val output = s"/avro/${date.toString(dayFormat)}"
  rmr(output)
  rdd.coalesce(64).map(x => (new AvroKey(x._1), x._2))
    .saveAsNewAPIHadoopFile(
    output,
    classOf[PageViewEvent],
    classOf[org.apache.hadoop.io.NullWritable],
    classOf[AvroKeyOutputFormat[PageViewEvent]],
    job.getConfiguration)

this works fine. I don't try to use AvroKeyValueOutputFormat anymore. But I  think now i would be able to. The key change was to use AvroKey and to set OutputKeySchema.


text:
Q: For which $r$ element of $(0,\infty$) $|x|^r \to 0$ if $x \to 0$ For which $r$ element of $(0,\infty$) $|x|^r \to 0$ if $x \to 0$?
Somehow I have no idea how to solve the problem. I know that it's true for $r\geq 1$.

A: For all $r>0$, one has
$$
\lim_{x\to 0}|x|^r=0,
$$ not only for $r\geq1$.


text:
Q: Firebase Find by specific field in url I want to find by specific field in give URL of firebase
I have seen you can find by following program:
new Firebase("https://examples-sql-queries.firebaseio.com/user")
.startAt('kato@firebase.com')
.endAt('kato@firebase.com')
.once('value', function(snap) {
console.log('accounts matching email address', snap.val())
});

but here we are not mentioning that find email address which matches'kato@firebase.com'. I want to mention that find only if email address matches

A: If you're looking to do a search of Firebase records, you'll need to pull in some other tool like ElasticSearch. See this blog post for more details: https://www.firebase.com/blog/2014-01-02-queries-part-two.html


text:
Q: Sending Files from GWT to a Web Server and Faking a Request IP I'm working on a project in GWT, however, I need to store the uploaded files on my personal web server.
The user will upload the files using GWT, my back end will store the files information in AppEngine's database and send the file to the server.
I'm thinking of creating a PHP script on the web server that will handle the files coming from GWT and sends the files as needed.
Also, I'm planning on validating the IP address of the request.
My concerns are:

*

*Is validating the IP address from which the request came good enough to ensure no one misuses the PHP script?

*How can GWT's server send the file to PHP?

*Is there a better way to do this?

Thanks in advance

A: The ip address can be verified using php's $_SERVER['REMOTE_ADDR'] .  This variable is pulled from Apache's TCP socket and it cannot be easily spoofed.  I personally have dug though the code to verify this.
However,  if the communication comes from the WIFI at a local cafe then you might have a problem.  An attacker on the network can sniff the connection and you are sharing your ip address with them.
The very secure method of protecting against this cafe scenario is using SSL.  HTTPS is easy to setup,  but make sure you buy a real certificate which should run you about ~$30 a year.


text:
Q: Splitting a dataframe into many smaller data frames evenly-ish I have the following frame called df which is 65 items long.
   Name  Data
0  Name1 Data1
1  Name2 Data2
2  Name3 Data3
....

I want to split it into 30 data frames as evenly as possible.
So with a length of 65, I want there to be 5 frames of length 3 and 25 of length 2 (which adds up to 65)
I use the following function:
def chunk(seq, size):
    return (seq[pos:pos + size] for pos in range(0, len(seq), size))

n = 30 #number of files

length = len(df)

counter=0

for df_chunk in chunk(frame, int(length / n) + (length % n > 0)):
    counter+=1
    df_chunk.to_csv(f"path/to/file{counter}.csv")

But I only get 21 files which are, 3 in length and 1 file which is 2 in length instead of 5 files which are 3 in length and 25 which are 2 in length.
Anyone has any ideas on how I can achieve what I want?

A: Use, np.array_split, from the documentation it says:

For an array of length l that should be split into n sections, it
  returns l % n sub-arrays of size l//n + 1 and the rest of size l//n.:

for counter, df_chunk in enumerate(np.array_split(df, 30), 1):
    df_chunk.to_csv(f"path/to/file{counter}.csv")


text:
Q: detecting that a file is currently being written to (I know It's a common problem but I couldn't find an exact answer)
I need to write a windows service that monitors a directory, and upon the arrival of a file, opens it, parses the text, does something with it and moves it to another directory afterwards. I used IsFileLocked method mentioned in this post to find out if a file is still been written. My problem is that I don't know how much it takes for another party to complete writing into the file. I could wait a few seconds before opening the file but this is not a perfect solution since I don't know in which rate is the file written to and a few seconds may not suffice.
here's my code:
while (true)
  {
    var d = new DirectoryInfo(path);
    var files = d.GetFiles("*.txt").OrderBy(f => f);

    foreach (var file in files)
    {
      if (!IsFileLocked(file))
      {
        //process file
      }
      else
      {
        //???
      }
    }
  }

A: I think you might use a FileSystemWatcher (more info about it here: http://msdn.microsoft.com/it-it/library/system.io.filesystemwatcher(v=vs.110).aspx ).
Specificially you could hook to the OnChanged event and after it raises you can check IsFileLocked to verify if it's still being written or not.
This strategy should avoid you to actively wait through polling.


text:
Q: Openpyxl - AttributeError: 'NoneType' object has no attribute 'lower' I'm currently trying to read the excel file, check if the quote category (third column on the Excel sheet) is "attitude", and if so, then print out the first two columns (which are the quote itself and the author).
I'm really stumped because even though I can see that category is indeed showing a proper value, the AttributeError is telling me otherwise. In addition, I checked the type of category and it says str.
I don't know that much about OOP, so I'm not sure if it's because there's an interference with something in the class itself.

Code:
from openpyxl import load_workbook

quotesTable = load_workbook(filename="Motivational Quotes Database.xlsx")

sheet = quotesTable.active
#print(sheet.title)

for row in sheet.iter_rows(min_row=5,min_col=1, max_col=3):
    category = row[2].value
    #print(category) - shows a value
    if category.lower() == 'attitude':
        print(row)

The website where I downloaded the excel sheet from: https://sharpquotes.com/download-45500-famous-motivational-quotes-database-in-excel-and-pdf/

A: As I suggested you need to check for empty cells and you also need to take into account any non-printing characters, e.g. line feeds.
from openpyxl import load_workbook

quotesTable = load_workbook(filename="Motivational Quotes Database.xlsx")

sheet = quotesTable.active
#print(sheet.title)

for row in sheet.iter_rows(min_row=5,min_col=1, max_col=3):
    category = row[2].value
    if  not category is None:
      if category.lower().startswith('attitude'):
        print(row)


text:
Q: Ejabberd Closes BOSH session after available presence I can't figure out why Ejabberd (14.07 & 14.12) is Closing a BOSH session after receiving an 'available' stanza for that same session.
I am using JsJac as web client lib and the session is being prebound server side and JsJac is doing 'inherit' on first page load and als on restart. The scenario in which this happens is after a successful connection and chat, i disable the network connection using chrome developer tools.
Ejabberd pings every 15 Seconds and is set to 'kill' on timeout. The Inactivity_timeout for BOSH in ejabberd.yml is set to 480.
The user can then click a link to try to reconnect. This is calling the exact same function as the initial prebind+client side inherit. After seemingly successful reconnect, a few seconds later, this:
Close session for
anonymous.c6502a950667fc79c3e1534559cc02e0@localhost/7f317c7fcbb9fb1b79c414c2d8afe05c
appears in the ejabberd log and a stanza of unavailable is sent to the other party in the chat.
Question: anybody have any idea why the BOSH session might be closed?
Here is the relevant piece of the Ejabberd log
2015-01-09 18:25:21.308 [debug] <0.2710.0>@ejabberd_receiver:process_data:343 Received XML on stream = <<"<presence from=\"anonymous.c6502a950667fc79c3e1534559cc02e0@jaxlecho.localhost\" type=\"available\"/>">>
2015-01-09 18:25:21.309 [debug] <0.2710.0>@shaper:update:117 State: {maxrate,50000,1899.4311619841992,1420824321273332}, Size=97
M=0.9887812185897407, I=35.836
2015-01-09 18:25:21.390 [info] <0.2709.0>@ejabberd_listener:accept:313 (#Port<0.41723>) Accepted connection ::FFFF:127.0.0.1:58483 -> ::FFFF:127.0.0.1:5280
2015-01-09 18:25:21.391 [debug] <0.5331.0>@ejabberd_http:init:145 S: [{[<<"presence">>],mod_webpresence},{[<<"captcha">>],ejabberd_captcha},{[<<"admin">>],ejabberd_web_admin},{[<<"http-bind">>],mod_http_bind},{[<<"http-poll">>],ejabberd_http_poll}]
2015-01-09 18:25:21.391 [info] <0.5331.0>@ejabberd_http:init:149 started: {gen_tcp,#Port<0.41723>}
2015-01-09 18:25:21.392 [debug] <0.5331.0>@ejabberd_http:process_header:277 (#Port<0.41723>) http query: 'POST' /http-bind
2015-01-09 18:25:21.392 [debug] <0.5331.0>@ejabberd_http:process_request:438 client data: <<"">>
2015-01-09 18:25:21.392 [debug] <0.5331.0>@ejabberd_http:process:350 [<<"http-bind">>] matches [<<"http-bind">>]
2015-01-09 18:25:21.393 [debug] <0.5331.0>@ejabberd_http:process:354 [{'Accept-Language',<<"en-US,en;q=0.8,es;q=0.6,und;q=0.4,nl;q=0.2,ca;q=0.2">>},{'Accept-Encoding',<<"gzip, deflate">>},{'Referer',<<"http://localhost:9080/">>},{'Accept',<<"*/*">>},{'Content-Type',<<"text/xml; charset=UTF-8">>},{'User-Agent',<<"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36">>},{<<"Origin">>,<<"http://localhost:9080">>},{'Content-Length',<<"158">>},{'Connection',<<"close">>},{'Host',<<"vagrant.localhost">>}]
2015-01-09 18:25:21.393 [debug] <0.5331.0>@mod_http_bind:process:68 Incoming data:
2015-01-09 18:25:21.393 [debug] <0.5331.0>@ejabberd_http_bind:parse_request:1128 --- incoming data ---

 --- END ---
2015-01-09 18:25:21.394 [debug] <0.5331.0>@ejabberd_http_bind:http_put:792 Looking for session: <<"a1e29cded4325f591bdc5b3646a8f3c78553cdd8">>
2015-01-09 18:25:21.394 [debug] <0.4052.0>@ejabberd_http_bind:handle_sync_event:440 New request: {http_put,876185,[{<<"xmlns">>,<<"http://jabber.org/protocol/httpbind">>},{<<"rid">>,<<"876185">>},{<<"sid">>,<<"a1e29cded4325f591bdc5b3646a8f3c78553cdd8">>},{<<"key">>,<<"38516e6b293e317ee957f6ef74fa23c68a003d19">>}],[],158,1,<<>>,{{0,0,0,0,0,65535,32512,1},58483}}
2015-01-09 18:25:21.395 [debug] <0.4052.0>@ejabberd_http_bind:handle_http_put_event:575 New request: {http_put,876185,[{<<"xmlns">>,<<"http://jabber.org/protocol/httpbind">>},{<<"rid">>,<<"876185">>},{<<"sid">>,<<"a1e29cded4325f591bdc5b3646a8f3c78553cdd8">>},{<<"key">>,<<"38516e6b293e317ee957f6ef74fa23c68a003d19">>}],[],158,1,<<>>,{{0,0,0,0,0,65535,32512,1},58483}}
2015-01-09 18:25:21.395 [debug] <0.4052.0>@ejabberd_http_bind:rid_allow:866 Previous rid / New rid: 876184/876185
2015-01-09 18:25:21.396 [debug] <0.4052.0>@ejabberd_http_bind:process_http_put:608 Actually processing request: {http_put,876185,[{<<"xmlns">>,<<"http://jabber.org/protocol/httpbind">>},{<<"rid">>,<<"876185">>},{<<"sid">>,<<"a1e29cded4325f591bdc5b3646a8f3c78553cdd8">>},{<<"key">>,<<"38516e6b293e317ee957f6ef74fa23c68a003d19">>}],[],158,1,<<>>,{{0,0,0,0,0,65535,32512,1},58483}}
2015-01-09 18:25:21.396 [debug] <0.4052.0>@ejabberd_http_bind:process_http_put:619 Key/OldKey/NextKey: 38516e6b293e317ee957f6ef74fa23c68a003d19/85e12ff1c2c9590328a5f0a1e58c14e52a8c6c42/85e12ff1c2c9590328a5f0a1e58c14e52a8c6c42
2015-01-09 18:25:21.397 [debug] <0.4052.0>@ejabberd_http_bind:process_http_put:659  -- SaveKey: 38516e6b293e317ee957f6ef74fa23c68a003d19
2015-01-09 18:25:21.397 [debug] <0.4052.0>@ejabberd_http_bind:process_http_put:671 reqlist: [{hbr,876185,<<"85e12ff1c2c9590328a5f0a1e58c14e52a8c6c42">>,[]},{hbr,876184,<<"85e12ff1c2c9590328a5f0a1e58c14e52a8c6c42">>,[{xmlstreamelement,{xmlel,<<"presence">>,[{<<"from">>,<<"anonymous.c6502a950667fc79c3e1534559cc02e0@jaxlecho.localhost">>},{<<"to">>,<<"agent1@localhost/cb_operator_1420818758533">>},{<<"type">>,<<"available">>},{<<"show">>,<<"chat">>}],[]}}]}]
2015-01-09 18:25:21.398 [debug] <0.4052.0>@ejabberd_http_bind:process_http_put:717 really sending now: []
2015-01-09 18:25:31.193 [debug] <0.2648.0>@ejabberd_router:do_route:322 route
    from {jid,<<>>,<<"localhost">>,<<>>,<<>>,<<"localhost">>,<<>>}
    to {jid,<<"agent1">>,<<"localhost">>,<<"cb_operator_1420818758533">>,<<"agent1">>,<<"localhost">>,<<"cb_operator_1420818758533">>}
    packet {xmlel,<<"iq">>,[{<<"id">>,<<"4293567110">>},{<<"type">>,<<"get">>}],[{xmlel,<<"ping">>,[{<<"xmlns">>,<<"urn:xmpp:ping">>}],[]}]}
2015-01-09 18:25:31.194 [debug] <0.2648.0>@ejabberd_local:do_route:296 local route
    from {jid,<<>>,<<"localhost">>,<<>>,<<>>,<<"localhost">>,<<>>}
    to {jid,<<"agent1">>,<<"localhost">>,<<"cb_operator_1420818758533">>,<<"agent1">>,<<"localhost">>,<<"cb_operator_1420818758533">>}
    packet {xmlel,<<"iq">>,[{<<"id">>,<<"4293"...>>},{<<"type">>,<<...>>}],[{xmlel,<<...>>,...}]}
2015-01-09 18:25:31.194 [debug] <0.2648.0>@ejabberd_sm:do_route:514 session manager
    from {jid,<<>>,<<"localhost">>,<<>>,<<>>,<<"localhost">>,<<>>}
    to {jid,<<"agent1">>,<<"localhost">>,<<"cb_operator_1420818758533">>,<<"agent1">>,<<"localhost">>,<<"cb_operator_1420818758533">>}
    packet {xmlel,<<"iq">>,[{<<"id">>,<<"4293"...>>},{<<"type">>,<<...>>}],[{xmlel,<<...>>,...}]}
2015-01-09 18:25:31.194 [debug] <0.2648.0>@ejabberd_sm:do_route:612 sending to process <0.4053.0>
2015-01-09 18:25:31.195 [debug] <0.5331.0>@ejabberd_http_bind:prepare_response:914 OutPacket: [{xmlstreamelement,{xmlel,<<"iq">>,[{<<"from">>,<<"localhost">>},{<<"to">>,<<"agent1@localhost/cb_operator_1420818758533">>},{<<"id">>,<<"4293567110">>},{<<"type">>,<<"get">>}],[{xmlel,<<"ping">>,[{<<"xmlns">>,<<"urn:xmpp:ping">>}],[]}]}}]
2015-01-09 18:25:31.195 [debug] <0.5331.0>@ejabberd_http_bind:send_outpacket:1056  --- outgoing data ---
<iq xmlns='jabber:client' from='localhost' to='agent1@localhost/cb_operator_1420818758533' id='4293567110' type='get'><ping xmlns='urn:xmpp:ping'/></iq>
 --- END ---
2015-01-09 18:25:31.228 [info] <0.2709.0>@ejabberd_listener:accept:313 (#Port<0.41753>) Accepted connection ::FFFF:127.0.0.1:58484 -> ::FFFF:127.0.0.1:5280
2015-01-09 18:25:31.230 [debug] <0.5332.0>@ejabberd_http:init:145 S: [{[<<"presence">>],mod_webpresence},{[<<"captcha">>],ejabberd_captcha},{[<<"admin">>],ejabberd_web_admin},{[<<"http-bind">>],mod_http_bind},{[<<"http-poll">>],ejabberd_http_poll}]
2015-01-09 18:25:31.231 [info] <0.5332.0>@ejabberd_http:init:149 started: {gen_tcp,#Port<0.41753>}
2015-01-09 18:25:31.231 [debug] <0.5332.0>@ejabberd_http:process_header:277 (#Port<0.41753>) http query: 'POST' /http-bind
2015-01-09 18:25:31.231 [debug] <0.5332.0>@ejabberd_http:process_request:438 client data: <<"<iq xmlns=\"jabber:client\" to=\"localhost\" id=\"4293567110\" type=\"result\"><ping xmlns=\"urn:xmpp:ping\"/></iq>">>
2015-01-09 18:25:31.231 [debug] <0.5332.0>@ejabberd_http:process:350 [<<"http-bind">>] matches [<<"http-bind">>]
2015-01-09 18:25:31.232 [debug] <0.5332.0>@ejabberd_http:process:354 [{'Accept-Language',<<"en-US,en;q=0.8,es;q=0.6,und;q=0.4,nl;q=0.2,ca;q=0.2">>},{'Accept-Encoding',<<"gzip, deflate">>},{'Referer',<<"http://localhost:9080/">>},{'Accept',<<"*/*">>},{'Content-Type',<<"text/xml; charset=UTF-8">>},{'User-Agent',<<"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36">>},{<<"Origin">>,<<"http://localhost:9080">>},{'Content-Length',<<"269">>},{'Connection',<<"close">>},{'Host',<<"vagrant.localhost">>}]
2015-01-09 18:25:31.233 [debug] <0.5332.0>@mod_http_bind:process:68 Incoming data: <iq xmlns="jabber:client" to="localhost" id="4293567110" type="result"><ping xmlns="urn:xmpp:ping"/></iq>
2015-01-09 18:25:31.233 [debug] <0.5332.0>@ejabberd_http_bind:parse_request:1128 --- incoming data ---
<iq xmlns="jabber:client" to="localhost" id="4293567110" type="result"><ping xmlns="urn:xmpp:ping"/></iq>
 --- END ---
2015-01-09 18:25:31.234 [debug] <0.5332.0>@ejabberd_http_bind:http_put:792 Looking for session: <<"a1e29cded4325f591bdc5b3646a8f3c78553cdd8">>
2015-01-09 18:25:31.234 [debug] <0.4052.0>@ejabberd_http_bind:handle_sync_event:440 New request: {http_put,876186,[{<<"xmlns">>,<<"http://jabber.org/protocol/httpbind">>},{<<"rid">>,<<"876186">>},{<<"sid">>,<<"a1e29cded4325f591bdc5b3646a8f3c78553cdd8">>},{<<"key">>,<<"9cffab80935471f419ce5c9201475df5935108d5">>}],[{xmlel,<<"iq">>,[{<<"xmlns">>,<<"jabber:client">>},{<<"to">>,<<"localhost">>},{<<"id">>,<<"4293567110">>},{<<"type">>,<<"result">>}],[{xmlel,<<"ping">>,[{<<"xmlns">>,<<"urn:xmpp:ping">>}],[]}]}],269,1,<<>>,{{0,0,0,0,0,65535,32512,1},58484}}
2015-01-09 18:25:31.234 [debug] <0.4052.0>@ejabberd_http_bind:handle_http_put_event:575 New request: {http_put,876186,[{<<"xmlns">>,<<"http://jabber.org/protocol/httpbind">>},{<<"rid">>,<<"876186">>},{<<"sid">>,<<"a1e29cded4325f591bdc5b3646a8f3c78553cdd8">>},{<<"key">>,<<"9cffab80935471f419ce5c9201475df5935108d5">>}],[{xmlel,<<"iq">>,[{<<"xmlns">>,<<"jabber:client">>},{<<"to">>,<<"localhost">>},{<<"id">>,<<"4293567110">>},{<<"type">>,<<"result">>}],[{xmlel,<<"ping">>,[{<<"xmlns">>,<<"urn:xmpp:ping">>}],[]}]}],269,1,<<>>,{{0,0,0,0,0,65535,32512,1},58484}}
2015-01-09 18:25:31.234 [debug] <0.4052.0>@ejabberd_http_bind:rid_allow:866 Previous rid / New rid: 876185/876186
2015-01-09 18:25:31.234 [debug] <0.4052.0>@ejabberd_http_bind:process_http_put:608 Actually processing request: {http_put,876186,[{<<"xmlns">>,<<"http://jabber.org/protocol/httpbind">>},{<<"rid">>,<<"876186">>},{<<"sid">>,<<"a1e29cded4325f591bdc5b3646a8f3c78553cdd8">>},{<<"key">>,<<"9cffab80935471f419ce5c9201475df5935108d5">>}],[{xmlel,<<"iq">>,[{<<"xmlns">>,<<"jabber:client">>},{<<"to">>,<<"localhost">>},{<<"id">>,<<"4293567110">>},{<<"type">>,<<"result">>}],[{xmlel,<<"ping">>,[{<<"xmlns">>,<<"urn:xmpp:ping">>}],[]}]}],269,1,<<>>,{{0,0,0,0,0,65535,32512,1},58484}}
2015-01-09 18:25:31.234 [debug] <0.4052.0>@ejabberd_http_bind:process_http_put:619 Key/OldKey/NextKey: 9cffab80935471f419ce5c9201475df5935108d5/38516e6b293e317ee957f6ef74fa23c68a003d19/38516e6b293e317ee957f6ef74fa23c68a003d19
2015-01-09 18:25:31.234 [debug] <0.4052.0>@ejabberd_http_bind:process_http_put:659  -- SaveKey: 9cffab80935471f419ce5c9201475df5935108d5
2015-01-09 18:25:31.234 [debug] <0.4052.0>@ejabberd_http_bind:process_http_put:671 reqlist: [{hbr,876186,<<"38516e6b293e317ee957f6ef74fa23c68a003d19">>,[]},{hbr,876185,<<"38516e6b293e317ee957f6ef74fa23c68a003d19">>,[{xmlstreamelement,{xmlel,<<"iq">>,[{<<"from">>,<<"localhost">>},{<<"to">>,<<"agent1@localhost/cb_operator_1420818758533">>},{<<"id">>,<<"4293567110">>},{<<"type">>,<<"get">>}],[{xmlel,<<"ping">>,[{<<"xmlns">>,<<"urn:xmpp:ping">>}],[]}]}}]}]
2015-01-09 18:25:31.235 [debug] <0.4052.0>@ejabberd_http_bind:process_http_put:717 really sending now: [{xmlel,<<"iq">>,[{<<"xmlns">>,<<"jabber:client">>},{<<"to">>,<<"localhost">>},{<<"id">>,<<"4293567110">>},{<<"type">>,<<"result">>}],[{xmlel,<<"ping">>,[{<<"xmlns">>,<<"urn:xmpp:ping">>}],[]}]}]
2015-01-09 18:25:31.235 [debug] <0.4053.0>@ejabberd_router:do_route:322 route
    from {jid,<<"agent1">>,<<"localhost">>,<<"cb_operator_1420818758533">>,<<"agent1">>,<<"localhost">>,<<"cb_operator_1420818758533">>}
    to {jid,<<>>,<<"localhost">>,<<>>,<<>>,<<"localhost">>,<<>>}
    packet {xmlel,<<"iq">>,[{<<"to">>,<<"localhost">>},{<<"id">>,<<"4293567110">>},{<<"type">>,<<"result">>}],[{xmlel,<<"ping">>,[{<<"xmlns">>,<<"urn:xmpp:ping">>}],[]}]}
2015-01-09 18:25:31.235 [debug] <0.4053.0>@ejabberd_local:do_route:296 local route
    from {jid,<<"agent1">>,<<"localhost">>,<<"cb_operator_1420818758533">>,<<"agent1">>,<<"localhost">>,<<"cb_operator_1420818758533">>}
    to {jid,<<>>,<<"localhost">>,<<>>,<<>>,<<"localhost">>,<<>>}
    packet {xmlel,<<"iq">>,[{<<"to">>,<<"loca"...>>},{<<"id">>,<<...>>},{<<...>>,...}],[{xmlel,<<...>>,...}]}
2015-01-09 18:25:31.570 [info] <0.5318.0>@ejabberd_c2s:terminate:1757 ({socket_state,ejabberd_http_bind,{http_bind,<0.5317.0>,{{0,0,0,0,0,65535,32512,1},58451}},ejabberd_http_bind}) Close session for anonymous.c6502a950667fc79c3e1534559cc02e0@localhost/2b8e1ac2921d6d3713396edff70a0d2a
2015-01-09 18:25:31.571 [debug] <0.5318.0>@mod_pubsub:node_action:5143 node_action <<"pubsub.localhost">> <<"flat">> get_entity_affiliations [<<"pubsub.localhost">>,{<<"anonymous.c6502a950667fc79c3e1534559cc02e0">>,<<"localhost">>,<<"2b8e1ac2921d6d3713396edff70a0d2a">>}]
2015-01-09 18:25:31.571 [debug] <0.5318.0>@mod_pubsub:node_call:5124 node_call <<"flat">> get_entity_affiliations [<<"pubsub.localhost">>,{<<"anonymous.c6502a950667fc79c3e1534559cc02e0">>,<<"localhost">>,<<"2b8e1ac2921d6d3713396edff70a0d2a">>}]
2015-01-09 18:25:31.571 [debug] <0.5318.0>@mod_pubsub:node_action:5143 node_action <<"pubsub.localhost">> <<"hometree">> get_entity_affiliations [<<"pubsub.localhost">>,{<<"anonymous.c6502a950667fc79c3e1534559cc02e0">>,<<"localhost">>,<<"2b8e1ac2921d6d3713396edff70a0d2a">>}]
2015-01-09 18:25:31.572 [debug] <0.5318.0>@mod_pubsub:node_call:5124 node_call <<"hometree">> get_entity_affiliations [<<"pubsub.localhost">>,{<<"anonymous.c6502a950667fc79c3e1534559cc02e0">>,<<"localhost">>,<<"2b8e1ac2921d6d3713396edff70a0d2a">>}]
2015-01-09 18:25:31.572 [debug] <0.5318.0>@mod_pubsub:node_action:5143 node_action <<"pubsub.localhost">> <<"pep">> get_entity_affiliations [<<"pubsub.localhost">>,{<<"anonymous.c6502a950667fc79c3e1534559cc02e0">>,<<"localhost">>,<<"2b8e1ac2921d6d3713396edff70a0d2a">>}]
2015-01-09 18:25:31.572 [debug] <0.5318.0>@mod_pubsub:node_call:5124 node_call <<"pep">> get_entity_affiliations [<<"pubsub.localhost">>,{<<"anonymous.c6502a950667fc79c3e1534559cc02e0">>,<<"localhost">>,<<"2b8e1ac2921d6d3713396edff70a0d2a">>}]
2015-01-09 18:25:31.573 [debug] <0.5318.0>@mod_carboncopy:disable:246 disabling for <<"anonymous.c6502a950667fc79c3e1534559cc02e0">>
2015-01-09 18:25:31.574 [debug] <0.5318.0>@mod_shared_roster:unset_presence:1081 unset_presence for <<"anonymous.c6502a950667fc79c3e1534559cc02e0">> @ <<"localhost">> / <<"2b8e1ac2921d6d3713396edff70a0d2a">> -> <<>> (0 resources)
2015-01-09 18:25:31.574 [debug] <0.5318.0>@ejabberd_router:do_route:322 route

More of the log is here. https://gist.github.com/Elexy/052a6a428fb57147bf4e

A: I solved my own question.

*

*I had set the mod_ping to KILL the session and I believe both the BOSH connection manager and mod_ping were killing sessions. I now changed mod_ping to 'timeout_action' to 'none' and let the BOSH connection manager handle the disconnection detection.

*The Inherit connection has to be "preloaded" with the parameters the BOSH client normally receives from the initial bosh response. In my case, relying on legacy code, the 'polling' paramater was set to 5 when the BOSH connection manager responds with 2.
JsJac was responding with up to 5 new requests within the 'requests' timeperiod of 2 seconds.
So my logic is this: mod_ping will send a ping every 15 seconds and the BOSH connection manager will check if the ping, or any other stanza is responded within 4 seconds.
Here is the initial response from the Ejabberd BOSH connection manager:
<stream:features xmlns:stream="http://etherx.jabber.org/streams"><bind xmlns="urn:ietf:params:xml:ns:xmpp-bind"/><session xmlns="urn:ietf:params:xml:ns:xmpp-session"/><sm xmlns="urn:xmpp:sm:2"/><sm xmlns="urn:xmpp:sm:3"/><c xmlns="http://jabber.org/protocol/caps" hash="sha-1" node="http://www.process-one.net/en/ejabberd/" ver="/nWL9StXSXhEsL2wg0+s4xo/UdA="/><register xmlns="http://jabber.org/features/iq-register"/></stream:features>

and my, now working, JS code:
   args.polling = 2;
   args.inactivity = 4;
   args.requests = 2;
   args.wait = 300;

   self.connection.inherit(args);

p.s. after reading  http://xmpp.org/extensions/xep-0124.html i set inactivity to 4 seconds in order to detect disconnections faster.


text:
Q: Read excel files from php I try to read excel files  with a PHPExcel library. Now I have a
problem, when upload a file with more of 1000 registers the velocity to insert this to Mysql Data Base is low can
takes more of 00:01:08 ( 1 min and 8 seconds). Can someone help me whit this issue??
Thanks for your help!


text:
Q: Environment Variables on Heroku and Mailgun Problems with Phoenix Framework I was following this guide on deploying to Heroku and this one for sending email.
Everything works fine in development. My variables are set in Heroku:
heroku config
...
MAILGUN_DOMAIN:  https://api.mailgun.net/v3/xxxxxx.mailgun.org
MAILGUN_KEY:     key-3-xxxxxx
...

And loaded from the config files like so:
config :take_two, Mailer,
    domain: System.get_env("MAILGUN_DOMAIN"),
    key: System.get_env("MAILGUN_KEY")

However when I try to send email on Heroku when the Mailgun config is set from environment variables I get this error:
** (FunctionClauseError) no function clause matching in IO.chardata_to_string/1
    (elixir) lib/io.ex:346: IO.chardata_to_string(nil)
    (elixir) lib/path.ex:467: Path.join/2
    (elixir) lib/path.ex:449: Path.join/1
             lib/client.ex:44: Mailgun.Client.send_without_attachments/2

This happens when the domain is not set for the Mailgun Client. But it is supposed to be set from the environment variable. I made a simple module to test:
defmodule TakeTwo.Mailer do
    require Logger

    use Mailgun.Client,
        Application.get_env(:take_two, Mailer)

    def blank_shot do
        Logger.info Application.get_env(:take_two, Mailer)[:domain]
        Logger.info Application.get_env(:take_two, Mailer)[:key]
        send_email from: "steve@xxx.com", to: "speggy@xxx.com", subject: "Hello", text: "This is a blank shot"
    end

When I run TakeTwo.Mailer.blank_shot I see the correct domain/key variables logged followed by the error. I am not sure how to debug the Mailgun client remotely.
Finally, if I recreate the above module in the shell (after running heroku run iex -S mix) it works just fine!?
I feel like when the original module is being loaded perhaps the environment variables have yet to be loaded??

A: The answer was a little buried in a comment so I wanted to make it easier to find. As the other answer mentions, the environment variables aren't available, but the buildpack lets you configure them to be:
I created a elixir_buildpack.config file and added the following:
config_vars_to_export=(DATABASE_URL MAILGUN_DOMAIN MAILGUN_KEY SECRET_KEY_BASE)

A: The environment variables aren't available at build time. I had the same issue and decided to get rid of the macro carrying the configuration. You can use this patch to move on.


text:
Q: Thowing an error in `await [].reduce(async ...)` crashes node I noticed today that if I throw an error inside of an await/async reduce, it will crash nodejs.
For example, this will crash my code:
    const tests = await [1, 2, 3].reduce(async i => {
        throw new Error("test error");
    }, Promise.resolve([] as any[]));

but the code code below is fine:
    const tests = [1, 2, 3].reduce(i => {
        throw new Error("test error");
    }, []);

or
    const tests = await Promise.all([1, 2, 3].map(async i => {
        throw new Error("test error");
    }));

Adding try/catch inside the reduce works well but sometimes I want reduce to stop processing and return the error like await Promise.all([].map(... does.
    const tests = await [1, 2, 3].reduce(async i => {
        try {
            throw new Error("test error");
        } catch(error) {
            console.log(error);
        }
    }, Promise.resolve([] as any[]));

I would like reduce to behave the same way map does when an error is thrown inside of it.
Is there a workaround for this?  An external reduce library?

A: Looks like it doesn't crash if you await the accumulator first.
This works as expected.
  const tests = await [1, 2, 3].reduce(async (acc, i) => {
        const result = await acc;
        throw new Error("test error"); // <--- error after await acc
        return result;
    }, Promise.resolve([] as any[]));

This crashes nodejs
  const tests = await [1, 2, 3].reduce(async (acc, i) => {
        throw new Error("test error"); // <--- error before await acc
        const result = await acc;
        return result;
    }, Promise.resolve([] as any[]));

I hope this helps someone.


text:
Q: Solving $\min\int_{a}^{1}t\dot{x}^2 dt$, with $x(a)=0$ and $x(1)=1$ We need to show that the problem above has a solution if $a\in(0,1)$, but not if $a=0$. We need to solve the problem with the Euler Equation
$$\frac{\partial F}{\partial x}-\frac{d}{dt}(\frac{\partial F}{\partial\dot{x}})=0$$
We define $F$ as
$$F(t,x,\dot{x})=t\dot{x}^2$$
Then subsequently we have:
$$ \begin{aligned} \frac{\partial F}{\partial x} &= 0 \\ \frac{\partial F}{\partial\dot{x}} &= 2t\dot{x} \end{aligned} $$
When substituting this into the above mentioned equation: $\frac{\partial F}{\partial x}-\frac{d}{dt}(\frac{\partial F}{\partial\dot{x}})=0$, my final equation becomes $0-\frac{d}{dt}(2t\dot{x}) = -2\dot{x}-2t\ddot{x} = 0$. This is probably where I make a mistake since the Student's Manual informs me that the Euler Equation is; $(d/dt)(t\dot{x})=0$. Can someone point out my mistake?


text:
Q: Compilation of IORef and STRef To measure the performance of those Refs, I dumped the assembly produced by GHC on the following code :
import Data.IORef

main = do
  r <- newIORef 18
  v <- readIORef r
  print v

I expected the IORef to be completely optimized away, leaving only a syscall to write stdout with string "18". Instead I get 250 lines of assembly. Do you know how many will actually be executed ? Here is what I think is the heart of the program :
.globl Main.main1_info
Main.main1_info:
_c1Zi:
    leaq -8(%rbp),%rax
    cmpq %r15,%rax
    jb _c1Zj
_c1Zk:
    movq $block_c1Z9_info,-8(%rbp)
    movl $Main.main2_closure+1,%ebx
    addq $-8,%rbp
    jmp stg_newMutVar#
_c1Zn:
    movq $24,904(%r13)
    jmp stg_gc_unpt_r1
.align 8
    .long   S1Zo_srt-(block_c1Z9_info)+0
    .long   0
    .quad   0
    .quad   30064771104
block_c1Z9_info:
_c1Z9:
    addq $24,%r12
    cmpq 856(%r13),%r12
    ja _c1Zn
_c1Zm:
    movq 8(%rbx),%rax
    movq $sat_s1Z2_info,-16(%r12)
    movq %rax,(%r12)
    movl $GHC.Types.True_closure+2,%edi
    leaq -16(%r12),%rsi
    movl $GHC.IO.Handle.FD.stdout_closure,%r14d
    addq $8,%rbp
    jmp GHC.IO.Handle.Text.hPutStr2_info
_c1Zj:
    movl $Main.main1_closure,%ebx
    jmp *-8(%r13)

I am concerned about this jmp stg_newMutVar#. It is nowhere else in the assembly, so maybe GHC resolves it at a later linking stage. But why is it even here and what does it do ? Can I dump the final assembly without any unresolved haskell symbols ?

A: Starting with a couple of links:

*

*The MutVar object definition.

*The cmm code for newMutVar.

*A non-comprehensive but helpful summary of GHC object layout.
The cmm and C sources aren't particularly readable if you're not already familiar with the macros and primops. Unfortunately, I don't know of a good way to view the assembly generated for cmm primops, short of looking into an executable with objdump or some other disassembler.
Still, I can summarize the runtime semantics of IORef.
IORef is a wrapper around MutVar# from GHC.Prim. As the doc says, MutVar# is like a single-element mutable array. It takes up two machine words, the first is the header, the second is the stored value (which is a pointer to a GHC object). A value of MutVar# is itself a pointer to this two-word object.
MutVar-s differ from normal immutable objects most notably by participating in a write barrier mechanism. GHC has generational garbage collection, so any MutVar that lives in an older generation must be also a GC root when collecting the younger generations, since mutating a MutVar may cause younger objects to become reachable. Therefore, whenever a MutVar is promoted from generation 0 (the youngest), it is added to a so-called "mutable list" that contains references to all such mutable objects. The mutable list gets rebuilt during GC of old generations. In short, MutVar-s in old generations are always present on the mutable list.
This is a rather simplistic way of dealing with mutable variables, and if we have large numbers of them in old generations, minor garbage collection slows down because of the bloated mutable list, and as a result the entire program slows down.
Since mutable variables aren't used prominently in production code, there hasn't been much demand or pressure for optimizing the RTS for their heavy usage.
If you need a large number of mutable variables, you should instead use a single mutable boxed array, because that's only a single reference on the mutable list and also has a bitmap-based optimization for GC traversal of elements that might have been mutated.
Also, as you see newMutVar# is only statically linked but not inlined, although it's a rather small chunk of code. As a result, it's also not optimized away. This is again broadly because of the lack of effort and attention for optimizing mutating code. By contrast, allocating and copying small known-sized primitive arrays is currently inlined and greatly optimized, because Johan Tibell who did large amount of work implementing the unordered-containers library made it so (in order to make unordered-containers faster).


text:
Q: Pipenv and setup.py workflow I am developing a package Pkg-1 in Python and my workflow is as follows:
I use pipenv to install a dependency A of Pkg-1 with pipenv install A and then I install Pkg-1 with pipenv install -e .
Now I want to use Pkg-1 in another package I develop, Pkg-2. I want to install Pkg-1 from github with a command like pipenv install ssh://git@github.com/user/Pkg1.git. At this point I need a setup.py in Pkg-1 with all the requirements fixed to the versions in Pipfile.lock. I create one with pipenv-setup.
The problem with this approach is the following:
Assume that Pkg-1 depends on the package A that depends on B with a restriction on the version of B >=3. This will install the latest version of B, assume it is 4. Running pipenv-setup will pin the version of B in the setup.py. Now you want to install a package C (as a dependency on Pkg-1) that depends on B with the restriction on the version <=3.5. Normally, pipenv would be able to resolve this dependency. But since the version of B is pinned for Pkg-1 in the setup.py, pipenv will complain.
Do you have any suggestions? Is there something wrong with my workflow?