text:
Q: Nutch with solr on https Good morning,
I come to you because I have a problem with Nutch (1.14) and Solr (7.2)
so it works fine until I put SSL in place.
With Solr in http, once the crawl is finished I execute this command
bin/nutch index -Dsolr.server.url=http://127.0.0.1:8983/solr/CORENAME crawltest/crawldb/ -linkdb crawltest/linkdb/ crawltest/segments/* -filter -normalize -deleteGone

And it works very well
However, once SSL is activated and the solr server in HTTPS, it is impossible to send the data to Solr.
I have added in nutch site the following properties
<name>solr.auth</name>
           <value>true</value>

<property>
           <name>solr.auth.username</name>
           <value>xxxx</value>

<property>
           <name>solr.auth.password</name>
           <value>xxxx</value>

property>
           <name>solr.server.type</name>
           <value>https</value>
property>
           <name>solr.server.url</name>
           <value>https://127.0.0.1:8983/solr/CORENAME</value>

But when I execute the previous command I get an error of this type
Caused by: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://127.0.0.1:8983/solr/CORENAME

&
caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

&
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Did you succeed in sending the data to a HTTPS solr?
Thanks
EDIT
To fix this errors following SSL procedure https://lucene.apache.org/solr/guide/7_0/enabling-ssl.html
And at the end execute this
keytool -import -file /path/to/solr/solr-ssl.pem -alias solr_cert -keystore /path/to/java-cacert (jre/lib/security/cacerts)
the default password is changeit

A: it advances a little, once the certificate import in cacerts I do not have any more this error.
Still in the same context, after activating SSL and authentication on the solr server. I use Nutch to Crawl the urls and send the data to solr.
Since the implementation of SSL I can no longer send data to SOLR.
When i execute this bin/nutch index -Dsolr.server.url=https://localhost:8983/solr/CORE -Dsolr.auth=true  -Dsolr.auth.username='solr'  -Dsolr.auth.password='xxxx' crawltest/crawldb/ -linkdb crawltest/linkdb/ crawltest/segments/* -filter -normalize -deleteGone
I have the following two errors:
java.lang.Exception: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://localhost:8983/solr/CORE: Expected mime type application/octet-stream but got text/html.

<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 401 Unauthorized</title>

HTTP ERROR 401

Problem accessing /solr/CORE/update. Reason:

    Unauthorized

        at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://localhost:8983/solr/CORE: Expected mime type application/octet-stream but got text/html.

<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 401 Unauthorized</title>

HTTP ERROR 401

Problem accessing /solr/CORE/update. Reason:

    Unauthorized

EDIT :
The first error is due to an authentication error.
After filling in the right values, I have a new error that I don't understand.. Do you have any ideas?
2018-06-20 09:47:18,116 INFO  regex.RegexURLNormalizer - can't find rules for scope 'indexer', using default
2018-06-20 09:47:19,151 INFO  indexer.IndexWriters - Adding org.apache.nutch.indexwriter.solr.SolrIndexWriter
2018-06-20 09:47:19,194 INFO  solr.SolrMappingReader - source: content dest: content
2018-06-20 09:47:19,194 INFO  solr.SolrMappingReader - source: title dest: title
2018-06-20 09:47:19,194 INFO  solr.SolrMappingReader - source: host dest: host
2018-06-20 09:47:19,194 INFO  solr.SolrMappingReader - source: segment dest: segment
2018-06-20 09:47:19,194 INFO  solr.SolrMappingReader - source: boost dest: boost
2018-06-20 09:47:19,195 INFO  solr.SolrMappingReader - source: digest dest: digest
2018-06-20 09:47:19,195 INFO  solr.SolrMappingReader - source: tstamp dest: tstamp
2018-06-20 09:47:19,525 INFO  solr.SolrIndexWriter - Indexing 250/250 documents
2018-06-20 09:47:19,525 INFO  solr.SolrIndexWriter - Deleting 0 documents
2018-06-20 09:47:19,808 INFO  solr.SolrIndexWriter - Indexing 250/250 documents
2018-06-20 09:47:19,809 INFO  solr.SolrIndexWriter - Deleting 0 documents
2018-06-20 09:47:19,951 WARN  mapred.LocalJobRunner - job_local146539832_0001
java.lang.Exception: java.io.IOException
    at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529)
Caused by: java.io.IOException
    at org.apache.nutch.indexwriter.solr.SolrIndexWriter.makeIOException(SolrIndexWriter.java:234)
    at org.apache.nutch.indexwriter.solr.SolrIndexWriter.push(SolrIndexWriter.java:213)
    at org.apache.nutch.indexwriter.solr.SolrIndexWriter.write(SolrIndexWriter.java:174)
    at org.apache.nutch.indexer.IndexWriters.write(IndexWriters.java:87)
    at org.apache.nutch.indexer.IndexerOutputFormat$1.write(IndexerOutputFormat.java:50)
    at org.apache.nutch.indexer.IndexerOutputFormat$1.write(IndexerOutputFormat.java:41)
    at org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.write(ReduceTask.java:493)
    at org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:422)
    at org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:369)
    at org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:57)
    at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
    at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://localhost:8983/solr/ESRF-EXTERNAL
    at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:589)
    at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:240)
    at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:229)
    at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
    at org.apache.nutch.indexwriter.solr.SolrIndexWriter.push(SolrIndexWriter.java:210)
    ... 16 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
    at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431)
    at sun.security.ssl.OutputRecord.write(OutputRecord.java:417)
    at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:886)
    at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:857)
    at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
    at org.apache.http.impl.io.AbstractSessionOutputBuffer.write(AbstractSessionOutputBuffer.java:181)
    at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:115)
    at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:146)
    at org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:96)
    at org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:112)
    at org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:117)
    at org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:265)
    at org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestEntity(ManagedClientConnectionImpl.java:203)
    at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:237)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:122)
    at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
    at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
    at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:481)
    ... 20 more
2018-06-20 09:47:20,873 ERROR indexer.IndexingJob - Indexer: java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:873)
    at org.apache.nutch.indexer.IndexingJob.index(IndexingJob.java:147)
    at org.apache.nutch.indexer.IndexingJob.run(IndexingJob.java:230)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.nutch.indexer.IndexingJob.main(IndexingJob.java:239)

Stacktrace
name    cpuTime / userTime

process reaper (37)
java.util.concurrent.SynchronousQueue$TransferStack@24197386
    1.8587ms
0.0000ms

process reaper (36)
java.util.concurrent.SynchronousQueue$TransferStack@24197386
    1.2672ms
0.0000ms

Scheduler-201556483 (31) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6202c0c1
    1.1534ms
0.0000ms

searcherExecutor-7-thread-1 (30)   java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@663d5859
    63.2030ms
50.0000ms

DestroyJavaVM (27)
    1164.4748ms
1040.0000ms

Thread-12 (25)
java.lang.Object@233fcafa
    0.1211ms
0.0000ms

Connection evictor (23)
    0.9319ms
0.0000ms

Connection evictor (22)
    2.0995ms
0.0000ms

org.eclipse.jetty.server.session.HashSessionManager@1a052a00Timer (21)    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6626f9cf
    4.2127ms
0.0000ms

qtp2012232625-20 (20)   java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@12d62902
    56.7955ms
50.0000ms

qtp2012232625-19 (19)
    47.6864ms
40.0000ms

qtp2012232625-18 (18) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@12d62902
    79.3320ms
70.0000ms

qtp2012232625-17 (17)   java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@12d62902
    100.9593ms
90.0000ms

qtp2012232625-16-acceptor-0@2d033cc4-ServerConnector@23c4c714{SSL,[ssl, http/1.1]}{0.0.0.0:8983} (16)
    4.5898ms
0.0000ms

qtp2012232625-15 (15)    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@12d62902
    73.3096ms
60.0000ms

qtp2012232625-14 (14)    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@12d62902
    18.7950ms
10.0000ms

qtp2012232625-13 (13)
    79.7804ms
70.0000ms

qtp2012232625-12 (12)
    70.2385ms
60.0000ms

qtp2012232625-11 (11)    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@12d62902
    22.1012ms
10.0000ms

ShutdownMonitor (10)
    0.3055ms
0.0000ms

Signal Dispatcher (5)
    0.0873ms
0.0000ms

Finalizer (3)
java.lang.ref.ReferenceQueue$Lock@1e254491
    8.2575ms
0.0000ms

Reference Handler (2)
java.lang.ref.Reference$Lock@431035b5
    6.3846ms
0.0000ms

EDIT2
To test I disabled authentication to see if the problem did not come from https. Without authentication it works!
I tried to change the file and include it in jetty-https.xml rather than jetty.xml.
I have 2 account configure like this
<security-constraint>
    <web-resource-collection>
      <web-resource-name>Solr authenticated application</web-resource-name>
      <url-pattern>/</url-pattern>
    </web-resource-collection>
    <auth-constraint>
      <role-name>admin</role-name>
    </auth-constraint>
  </security-constraint>
  <login-config>
    <auth-method>BASIC</auth-method>
    <realm-name>Test Realm</realm-name>
  </login-config>

security.json
{
"authentication":{
   "blockUnknown": true,
   "class":"solr.BasicAuthPlugin",
   "credentials":{"solr":"xxxx"}
},
"authorization":{
   "class":"solr.RuleBasedAuthorizationPlugin",
   "permissions":[{"name":"security-edit",
      "role":"admin"}],
   "user-role":{"solr":"admin"}
}}

when i execute the following command
bin/nutch index -Dsolr.server.url=https://localhost:8983/solr/MYCORE -Dsolr.auth=true  -Dsolr.auth.username='admin'  -Dsolr.auth.password='xxxx' crawltest/crawldb/ -linkdb crawltest/linkdb/ crawltest/segments/* -filter -normalize -deleteGone

and I get this error
java.lang.Exception: java.io.IOException
        at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529)
Caused by: java.io.IOException
        at org.apache.nutch.indexwriter.solr.SolrIndexWriter.makeIOException(SolrIndexWriter.java:234)
        at org.apache.nutch.indexwriter.solr.SolrIndexWriter.push(SolrIndexWriter.java:213)
        at org.apache.nutch.indexwriter.solr.SolrIndexWriter.write(SolrIndexWriter.java:174)
        at org.apache.nutch.indexer.IndexWriters.write(IndexWriters.java:87)
        at org.apache.nutch.indexer.IndexerOutputFormat$1.write(IndexerOutputFormat.java:50)
        at org.apache.nutch.indexer.IndexerOutputFormat$1.write(IndexerOutputFormat.java:41)
        at org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.write(ReduceTask.java:493)
        at org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:422)
        at org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:369)
        at org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:57)
        at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
        at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://localhost:8983/solr/MYCORE
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:589)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:240)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:229)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
        at org.apache.nutch.indexwriter.solr.SolrIndexWriter.push(SolrIndexWriter.java:210)
        ... 16 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
        at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431)
        at sun.security.ssl.OutputRecord.write(OutputRecord.java:417)
        at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:886)
        at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:857)
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
        at org.apache.http.impl.io.AbstractSessionOutputBuffer.write(AbstractSessionOutputBuffer.java:181)
        at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:115)
        at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:146)
        at org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:96)
        at org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:112)
        at org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:117)
        at org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:265)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestEntity(ManagedClientConnectionImpl.java:203)
        at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:237)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:122)
        at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:481)
       ... 20 more
2018-06-25 09:38:41,870 ERROR indexer.IndexingJob - Indexer: java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:873)
        at org.apache.nutch.indexer.IndexingJob.index(IndexingJob.java:147)
        at org.apache.nutch.indexer.IndexingJob.run(IndexingJob.java:230)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.nutch.indexer.IndexingJob.main(IndexingJob.java:239)

And when i execute this
bin/nutch index -Dsolr.server.url=https://localhost:8983/solr/MYCORE -Dsolr.auth=true  -Dsolr.auth.username='solr'  -Dsolr.auth.password='xxxxx' crawltest/crawldb/ -linkdb crawltest/linkdb/ crawltest/segments/* -filter -normalize -deleteGone

I get this error
java.lang.Exception: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://localhost:8983/solr/MYCORE: Expected mime type application/octet-stream but got text/html.

<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 401 Unauthorized</title>

HTTP ERROR 401

Problem accessing /solr/MYCORE/update. Reason:

    Unauthorized

        at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://localhost:8983/solr/MYCORE: Expected mime type application/octet-stream but got text/html.

<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<title>Error 401 Unauthorized</title>

HTTP ERROR 401

Problem accessing /solr/MYCORE/update. Reason:

    Unauthorized

at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:544)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:240)
        at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:229)
        at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
        at org.apache.nutch.indexwriter.solr.SolrIndexWriter.push(SolrIndexWriter.java:210)
        at org.apache.nutch.indexwriter.solr.SolrIndexWriter.write(SolrIndexWriter.java:174)
        at org.apache.nutch.indexer.IndexWriters.write(IndexWriters.java:87)
        at org.apache.nutch.indexer.IndexerOutputFormat$1.write(IndexerOutputFormat.java:50)
        at org.apache.nutch.indexer.IndexerOutputFormat$1.write(IndexerOutputFormat.java:41)
        at org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.write(ReduceTask.java:493)
        at org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:422)
        at org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:369)
        at org.apache.nutch.indexer.IndexerMapReduce.reduce(IndexerMapReduce.java:57)
        at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
        at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2018-06-25 09:45:20,106 ERROR indexer.IndexingJob - Indexer: java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:873)
        at org.apache.nutch.indexer.IndexingJob.index(IndexingJob.java:147)
        at org.apache.nutch.indexer.IndexingJob.run(IndexingJob.java:230)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.nutch.indexer.IndexingJob.main(IndexingJob.java:239)

or now this :
java.lang.Exception: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:8983/solr/MYCORE: Expected mime type application/octet-stream but got text/html.

<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 503 </title>

HTTP ERROR: 503

Problem accessing /solr/MYCORE/update. Reason:

    Service Unavailable

        at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:8983/solr/MYCORE: Expected mime type application/octet-stream but got text/html.

<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 503 </title>

HTTP ERROR: 503

Problem accessing /solr/MYCORE/update. Reason:

    Service Unavailable

Solr logs :
2018-06-25 14:18:44.352 INFO  (main) [   ] o.e.j.s.Server jetty-9.3.20.v20170531
2018-06-25 14:18:44.597 WARN  (main) [   ] o.e.j.w.WebAppContext Failed startup of context o.e.j.w.WebAppContext@5891e32e{/solr,file:///app/solr-7.2.1/server/solr-webapp/webapp/,UNAVAILABLE}{/app/solr-7.2.1/server/solr-webapp/webapp}
java.lang.IllegalStateException: No LoginService for org.eclipse.jetty.security.authentication.BasicAuthenticator@64c87930 in org.eclipse.jetty.security.ConstraintSecurityHandler@400cff1a
        at org.eclipse.jetty.security.authentication.LoginAuthenticator.setConfiguration(LoginAuthenticator.java:76)
        at org.eclipse.jetty.security.SecurityHandler.doStart(SecurityHandler.java:354)
        at org.eclipse.jetty.security.ConstraintSecurityHandler.doStart(ConstraintSecurityHandler.java:448)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
        at org.eclipse.jetty.server.handler.ScopedHandler.doStart(ScopedHandler.java:120)
        at org.eclipse.jetty.server.session.SessionHandler.doStart(SessionHandler.java:116)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
        at org.eclipse.jetty.server.handler.ScopedHandler.doStart(ScopedHandler.java:120)
        at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:809)
        at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:345)
        at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1406)
        at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1368)
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:778)
        at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262)
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:522)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:41)
        at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
        at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:499)
        at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:147)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180)
        at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:458)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
        at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610)
        at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529)
        at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392)
        at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:561)
        at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:236)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
 at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
        at org.eclipse.jetty.server.Server.start(Server.java:422)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
        at org.eclipse.jetty.server.Server.doStart(Server.java:389)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1520)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1442)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.eclipse.jetty.start.Main.invokeMain(Main.java:215)
        at org.eclipse.jetty.start.Main.start(Main.java:458)
        at org.eclipse.jetty.start.Main.main(Main.java:76)
2018-06-25 14:18:44.745 INFO  (main) [   ] o.e.j.s.Server Started @799ms

I Solved "No LoginService" by move the security.json on /var/solr/data/ (SOLR_HOME)
EDIT3:
now I only get an error message "No allow" when I want to send nutch data to solr. Also I can no longer connect to the admin interface I get the same error. I think it came from the security.json file
{
"authentication":{
   "class":"solr.BasicAuthPlugin",
   "credentials":{"solr":"xxxxxx"}
},
"authorization":{
   "class":"solr.RuleBasedAuthorizationPlugin"
   "permissions":[{"name":"security-edit","role":"adminRole"},{"name":"collection-admin-edit","role":"adminRole"},{"name":"update","role":"adminRole"},{"name":"all","role":"adminRole"},{"name":"core-admin-edit","role":"adminRole"},{"name":"read","role":"adminRole"},{"name":"config-edit","role":"adminRole"},{"name":"core-admin-read","role":"adminRole"},{"name":"core-admin-read","role":"adminRole"}]
   "user-role":{"solr":"adminRole"}
}}

What did I do wrong? thanks

A: I added a new answer because the previous was too long
SOLVED API AUTHENTICATION BUT NOT WITH NUTCH:
So to make authentication work via the API I deleted the configuration made in jetty-https.xml, webdefault.xml and I deleted the realm.properties file as well as the basic auth options in solr.in.sh
I only work on the security.json file in SOLR HOME
In fact the biggest problem was that I did not use an encrypted password to test the connection while without it impossible to connect. On the other hand I still have the problem via nutch which is not allowed.
Here is the security.json file
`{
    "authentication":{
        "class":"solr.BasicAuthPlugin",
        "credentials":{
            "solr":"hzMjhfgN4b9X8KR0QgLB2Um3cUzqDzJygtEBL/O7g5E= CkP7HyXjYvqKNF3F4hBjnVvKGQOkLc/ta4FaNIkqgII="
        }
    },
    "authorization":{
        "class":"solr.RuleBasedAuthorizationPlugin",
        "permissions":[
            {
                "name":"security-edit",
                "role":"adminRole"
            },
            {
                "name":"collection-admin-edit",
                "role":"adminRole"
            },
            {
                "name":"update",
                "role":"adminRole"
            },
            {
                "name":"config-edit",
                "role":"adminRole"
            },
            {
                "name":"core-admin-edit",
                "role":"adminRole"
            },
            {
                "name":"core-admin-read",
                "role":"adminRole"
            {
                "name":"schema-edit",
                "role":"adminRole"
            },
            {
                "name":"all",
                "role":"adminRole"
            }
        ],
        "user-role":{
            "solr":"adminRole"
        }
    }
}
`

The encrypted password represented value "test"
To test the file code i suggest this http://json.parser.online.fr/
What could I have missed to be able to update solr with nutch?
SOLVED
Add in update role path for dataimport
           {
                "name":"update",
                "path":"/dataimport",
                "role":"adminRole"
            },

But now i can index nutch to solr but i have a new error when i'm crawling...
`Thu Jun 28 09:21:03 CEST 2018 : Iteration 2 of 5
Generating a new segment
/app/nutch-external/bin/nutch generate -D mapreduce.job.reduces=2 -D mapred.child.java.opts=-Xmx1000m -D mapreduce.reduce.speculative=false -D mapreduce.map.speculative=false -D mapreduce.map.output.compress=true crawl//crawldb crawl//segments -topN 50000 -numFetchers 1 -noFilter
Generator: starting at 2018-06-28 09:21:04
Generator: Selecting best-scoring urls due for fetch.
Generator: filtering: false
Generator: normalizing: true
Generator: topN: 50000
Generator: 0 records selected for fetching, exiting ...
Generate returned 1 (no new segments created)
Escaping loop: no more URLs to fetch now
`

And during the first Iteration i have this errors
`Authorization challenge processed
No form element found with 'id' = adminRole, trying 'name'.
No form element found with 'id' = adminRole, trying 'name'.
No form element found with 'name' = adminRole
No form element found with 'name' = adminRole
Supported authentication schemes in the order of preference: [ntlm, digest, basic]
Supported authentication schemes in the order of preference: [ntlm, digest, basic]
Challenge for ntlm authentication scheme not available
Challenge for ntlm authentication scheme not available
Challenge for digest authentication scheme not available
basic authentication scheme selected
Using authentication scheme: basic
Authorization challenge processed
No form element found with 'id' = adminRole, trying 'name'.
No form element found with 'name' = adminRole
Failed to get protocol output
java.lang.RuntimeException: java.lang.IllegalArgumentException: No form exists: adminRole
    at org.apache.nutch.protocol.httpclient.Http.resolveCredentials(Http.java:506)
    at org.apache.nutch.protocol.httpclient.Http.getResponse(Http.java:183)
    at org.apache.nutch.protocol.http.api.HttpBase.getProtocolOutput(HttpBase.java:276)
    at org.apache.nutch.fetcher.FetcherThread.run(FetcherThread.java:342)
Caused by: java.lang.IllegalArgumentException: No form exists: adminRole
    at org.apache.nutch.protocol.httpclient.HttpFormAuthentication.getLoginFormParams(HttpFormAuthentication.java:219)
    at org.apache.nutch.protocol.httpclient.HttpFormAuthentication.login(HttpFormAuthentication.java:95)
    at org.apache.nutch.protocol.httpclient.Http.resolveCredentials(Http.java:504)
    ... 3 more
Challenge for digest authentication scheme not available
basic authentication scheme selected
Using authentication scheme: basic
Authorization challenge processed
No form element found with 'id' = adminRole, trying 'name'.
`

It's still security.json file. Have you any ideas ? thanks
SOLVED
I solved this by configure httpclient-auth.xml in /nutch/conf/
<auth-configuration>
   <credentials username="solr" password="xxxxx">
      <authscope host="localhost" port="8983"/>
   </credentials>
</auth-configuration>

Thanks for your help


text:
Q: How to call a return value from a function into another function that has already a parameter? I have $result which contains my query to perform this task, it works fine and I dont have any problem with it but I want to use the return value of the first function (buildRatingSystem) in my second function in order to perform the task, could anyone help me how to do it cause I have tried bucket of things but nothing is working. I was thinking of making $average as a global variable but nothing worked so please need help. Thanks ahead
function buildRatingSystem( array $result): string
{
$ratingNumber = 0;
$count = 0;
// Initialize Global variables in order to be used in other parts of the website
$GLOBALS['fiveStarRating'] = 0;
$GLOBALS['fourStarRating'] = 0;
$GLOBALS['threeStarRating'] = 0;
$GLOBALS['twoStarRating'] = 0;
$GLOBALS['oneStarRating'] = 0;

foreach($result as $rate) {
$ratingNumber+= $rate['ratingNumber'];
$count += 1;
if($rate['ratingNumber'] == 5) {
$GLOBALS['fiveStarRating'] +=1;
}
else if($rate['ratingNumber'] == 4) {
$GLOBALS['fourStarRating'] +=1;
}
else if($rate['ratingNumber'] == 3) {
$GLOBALS['threeStarRating'] +=1;
}
else if($rate['ratingNumber'] == 2) {
$GLOBALS['twoStarRating'] +=1;
}
else if($rate['ratingNumber'] == 1) {
$GLOBALS['oneStarRating'] +=1;
}
;
}
return $average= $ratingNumber/$count ;
}

This is the second function where i want to use value in $average
function buildProductsDisplay(array $products): string
{
foreach ($products as $product) {
$html .= '
';
$averageRating = round($average, 0);
for ($i = 1; $i <= 5; $i++) {
$ratingClassStar = "fa-star-o";
if($i <= $averageRating) {
$ratingClassStar = "fa-star";
}
$html .="";
}
$html .= '
';
}
return $html;
}

redpajama_set_name: RedPajamaStackExchange


text:
Q: Can you mask EditText values on a char per char basis? I have an EditText where I want to mask character values but hyphens. For example "11-11" would ideally display as "**-**".
I know if you define android:password="true" in your EditText configuration, it will mask all the characters (except the last entered value). Is there a way to override that functionality to display hyphens in clear text?
*edit - Moved answer out of question...

A: I figured this out if anyone in the future wants to know how to do it. I extended PasswordTransformationMethod, and overrode getTransformation(). Unfortunately, the authors of that class did not make it the slightest bit extensible, so I ended up having to copy in all the code for the private classes PasswordCharSequence and Visible.
The actual change to allow hyphens was simple. I replaced:
if (!visible) {
    dest[i - start + off] = DOT;
}

with:
if (!visible && dest[i - start + off] != '-') {
    dest[i - start + off] = DOT;
}

Hope that helps someone. Also, I'd be interested if someone finds a cleaner solution to this problem.
*Edit - Actually, I just noticed that the masking doesn't work in landscape orientation... I'll update this when I figure it out.


text:
Q: Structuring database table with large text field I'm looking for an advice about structuring a data table as in title to make it efficient for querying and writing. I store information about an entity which has usual data types, numbers, short string etc. Now I need to store additional field with large amount of data (~ 30 KB) and I'm looking at two options:

*

*add a column an nvarchar(100000) in the entity table

*create separate table to store such data and link from the entity table

other factors:

*

*each entity row will have an accompanying large text field

*each accompanying text field will have at least 20 KB of data

*~20% of queries against entity table also need the large field. Other queries can do without it

*~95% of queries seek for single entity

*I'm using an O/RM to access the data, so all the columns are pulled in (I could pick and choose by making the code look horrid)

Right now I'm leaning toward having a separate table, but it also has a bad side in that I have to remember some concerns about keeping data consistent.
It's hard to make a decision without doing a real benchmark, but this could require few days of work so I'm turning to SO for a shortcut.

A: We recently had this exact problem. (though it was an XML Column instead of an NVarchar(max)) but the problem is the exact same.
Our use case was to display a list of records on a web page (the first 6 columns) of the table
and then to store a tonne of additional information in the nvarchar(max) column which got displayed once your selected an individual row.
Originally a single table contained all 7 columns.
TABLE 1
    INT ID (PK IDentity)
    5 other columns
    NVARCHAR(max)

Once we refactored it to the following we got a massive perf. boost.
TABLE 1
    INT ID (PK IDentity)
    5 other columns
    INT FID  (FK -TABLE2)

TABLE 2
    FID (PK IDENTITY)
    nvarchar(max)

The reason is that if the nvarchar(max) is short enough, it will be stored "in-row" but if it extends beyond the page size, then it gets stored elsewhere, and depending on a) the size of the table and record set your querying, and b) the amount of data in your nvarchar(max) this can have a pretty dramatic perf. drop.
Have a read of this link:
http://msdn.microsoft.com/en-us/library/ms189087.aspx

When a large value type or a large object data type column value is
  stored in the data row, the Database Engine does not have to access a
  separate page or set of pages to read or write the character or binary
  string. This makes reading and writing the in-row strings about as
  fast as reading or writing limited size varchar, nvarchar, or
  varbinary strings. Similarly, when the values are stored off-row, the
  Database Engine incurs an additional page read or write.

I'd bite the bullet now, and design your tables to store the large nvarchar(max) in a seperate table, assuming you don't need the data it contains in every select query
With regards, your comment about using an ORM. we were also using NHibernate in our situation. It's relatively easy to configure your mappings to lazy-load the related object on demand.

A: Well, you could start with documentation...

add a column an nvarchar(100000) in the entity table

Given the documented max size of 8000 bytes for a field and thus nvarchar(4000) being the maximum, I am interested to know how you consider this an option?
nvarchar(max) - ntext etc. would be the right thing to do.
And then you should read up on full text search, which is in SQL Serve pretty much for ages. Your ORM likely does not support it though - technology choices limiting features is typical when people - have a problem abstract things. Not something I would access with an ORM.


text:
Q: How to extract from dictionary to another dictionary, based on condition? Input :
t1_dict = {"k1":{"api1":200,"api2":500,"api3":200},"k2":{"api1":500,"api2":200,"api3":500}}

Output:
t2_dict = {'k2': {'api'1: 500,'api3':500}, 'k1': {'api2': 500}}

Need to extract whose values not equal 200 to another dictionary.
I could able to do it, as below.
t2_dict = {}
   for k1,v1 in api_status.items():
      t2_dict[k1] = {}
      for k2, v2 in v1.items():
         if v2 != 200:
            t2_dict[k1][k2] = v2

Can it be done in better way ? Any one-liner or other best solution ? Any optimisation ?

A: Alternative one-liner (after the setup):
#Value to exclude
Value=200

t1_dict={"k1":{"api1":200,"api2":500,"api3":200},"k2":{"api1":500,"api2":200,"api3":500}}

t2_dict = {j:{k:v for k,v in t1_dict[j].items() if v != Value} for j in t1_dict}

print(t2_dict)

Credit to the top rated solution found here for the root of the idea, I simply nested it inside another loop: Remove a dictionary key that has a certain value
Hope this helps!


text:
Q: sql query inside controller returns null I am trying to generate a value first and then I try to pass that value into SQL query but my query returns null. I tried debugging my code I have value inside myvalue variable.
I am not sure it can be done the way I am doing. Can pass I the value like below? if not any suggestions, please.
public ActionResult Index(View model)
{
    using (var con = new SqlConnection(ConfigurationManager.ConnectionStrings["DB_MYTABLE_CONNECTION_STRING"].ConnectionString))
    {
        var myvalue = MyUtil.GenerateKey(model.Name, model.phonenumber);

        con.Open();
        string query = "select count(*) from customer where key = @myvalue";

        using (var cmd = new SqlCommand(query, con))
        {
            int rowsAmount = (int)cmd.ExecuteScalar();
        }
    }
}

A: You need to pass the SqlCommand the parameter name and value:
public ActionResult Index(View model)
{
    using (var con = new SqlConnection(ConfigurationManager.ConnectionStrings["DB_MYTABLE_CONNECTION_STRING"].ConnectionString))
    {

        var myvalue = MyUtil.GenerateKey(model.Name, model.phonenumber);

        con.Open();
        string query = "select count(*) from customer where  key=@myvalue";

        using (var cmd = new SqlCommand(query, con))
        {
            cmd.Parameters.Add("@myvalue", myvalue);
            int rowsAmount = (int)cmd.ExecuteScalar();
        }
    }
}

A: The ActionResult Index(View Model)
says you are passing in the model to the Controller -- if you have the data in the Model, you should do your lookup in the Model and have the result as part of your View Model
Data Access in my code is usually by database in its class within the Models - and ANYTHING that hits the database is usually in a stored procedure in the database - to avoid SQL injection. Create your View Model to pass the right data for the controller to pass to the view.
Test your stored proc in the database and then use in a Data Access Layer, and store it into a Model class to inject it into your controllers.


text:
Q: Symfony/PHPUnit enable profiler in tests I want to enable the symfony profiler for my tests,I followed this documentation but my profiler stay empty. I'm using PHPUnit 7.5.16, and Symfony 4.3.4 with PHP 7.2.19.
What should I add ?

#web_profiler:
#    toolbar: false
#    intercept_redirects: false
#
#framework:
#    profiler: { collect: false }

web_profiler:
  toolbar: true
  intercept_redirects: false

framework:
#  profiler: { only_exceptions: false }
  # make tests slowest, but it s usefull in local,watch this doc: https://symfony.com/doc/current/testing/profiling.html
  profiler: { enabled: true, collect: true }

and in the setUp() method I have
parent::setUp();
$client = static::createClient();
$client->enableProfiler();

A: You'll likely also need:
framework:
    test: true

Since you are enabling the profile in the test, you don't need to do so with
framework:
   profiler: { enabled: true, ... # can be 'enabled: false', if you enable it in a test

My own config for the test environment is:
> bin/console -e test debug:config framework profiler

Current configuration for "framework.profiler"
==============================================

enabled: true
collect: false
only_exceptions: false
only_master_requests: false
dsn: 'file:%kernel.cache_dir%/profiler'

and framework: test: true. With this config, I've got a test environment that enables the profile, and checks for results in it.


text:
Q: I have problems when mounting my USB drive in a nonstandard location My USB drive gets mounted under /media/username. I tried using umount to mount under /media/usbdisk instead. Well, theoretically it worked, but when I used MultiSystem Multiboot USB Creator it doesn't recognize my USB drive under /media. I searched this entire site and didn't find a solution. This issue also happened with the Sardu Multiboot USB Creator. And when I tried YUMI, it also did not recognize the drive. However, my OS does correctly recognise the USB drive under /media/usbdisk.
I am using Ubuntu 14.04.3 LTS and my kernel version is 3.19.0-26. How do I solve this? I could find only partial solutions to this problem.

A: It may be confusing that bootable USB creators don't recognize the USB drive when the OS is doing it properly. The simple solution is to use Gnome's own disk utility. Use Startup Disk Creator and format your drive (clicking "erase disk" will do the job for you). Then you are good to go.
P.S.: I don't recommend GParted 0.18 because it completely trashed my USB drive. It has problems with FAT32.


text:
Q: List lookup to list column in other Site Collection I have an existing SharePoint custom list with various columns. I require an additional column that is a lookup to a list in another site collection which I'm aware is not possible OOTB. How can I achieve this? After searching the web I have only seen options to get information from another site collection and put it in a web part, but I need the lookup to be a list column.
I'm aware this will likely require some form of customizations / custom code - please go easy on me with easy to follow instructions as I've never delved into customizations before.
I'm using SharePoint 2013.

A: I've addressed similar situations creating External Content Types that populate my external lists from another sources (another site, another application and even another databases) that means that you need to have de BCS Service running and to develop a little visual studio solution to define which columns will be synced, you will need SP Designer too.
Start Here:
External content types in SharePoint 2013

A: Have you looked at utilizing the content type hub feature? This capability tied to the Managed Metadata service allows you to identify a site as a publishing hub that then sends its published content  types and respective columns to all sites that subscribe to it. You could ultimately switch your lookup column to a simpler type (I.e. choice) or you could avoid all of this together if you chose to store your values directly in the global term store that is available to all sites. Both the content type hub and global term store are solid choices to centralize a list of options that should be consistent across all sites. Please feel free to let me know if this is of interest to you, but you're not quite sure how to tackle, as I'm happy to assist.
Cheers,
Matt


text:
Q: Error while installing Nitrogen Thought I'd explore Nitrogen. Following install instructions outlined in http://nitrogenproject.com/doc/tutorial.html, I pulled source from GitHub, executed second step, make rel_inets, and hit a wall. Here is what the system told me:
nitrogen$ make rel_inets
...
make[2]: Leaving directory `/home/lloyd/Erl/nitrogen'
Running Erlang
Generating "reltool.config" with merged overlays
...Loading base file: "reltool_base.config"
...Merging Overlay from "reltool_inets.config"
...Writing "reltool.config"
...SUCCESS
make[2]: Entering directory `/home/lloyd/Erl/nitrogen'
Running Erlang
==> rel (generate)
ERROR: generate failed while processing /home/lloyd/Erl/nitrogen/rel: {'EXIT',      {{badmatch,{error,"Illegal library \"/home/lloyd/Programming/Erlang/zippity/apps\": no such file or directory"}},

Turns out that /home/lloyd/Programming/Erlang/zippity/apps was deleted long ago; e.g. it's not in my directory tree.
Can any kind wizards tell me why in the world would make dredge up this ancient history? And... what can I do about it?
Many thanks,
LRP
As requested, I've edited line 231 of Makefile. Here's output of make rel_inets:
lloyd@Reliance:~/Erl/nitrogen$ make rel_inets
make[1]: Entering directory `/home/lloyd/Erl/nitrogen'
./rebar get-deps
Running Erlang
==> rel (get-deps)
==> nitrogen (get-deps)
./rebar compile
Running Erlang
==> rel (compile)
==> nitrogen (compile)
make[2]: Entering directory `/home/lloyd/Erl/nitrogen'
make[2]: Leaving directory `/home/lloyd/Erl/nitrogen'
Running Erlang
Generating "reltool.config" with merged overlays
...Loading base file: "reltool_base.config"
...Merging Overlay from "reltool_inets.config"
...Writing "reltool.config"
...SUCCESS
make[2]: Entering directory `/home/lloyd/Erl/nitrogen'
Running Erlang
DEBUG: Consult config file "/home/lloyd/Erl/nitrogen/rel/rebar.config"
DEBUG: Rebar location: "/home/lloyd/Erl/nitrogen/rel/rebar"
DEBUG: is_rel_dir(/home/lloyd/Erl/nitrogen/rel) -> {true,
                                                    "/home/lloyd/Erl/nitrogen/rel/reltool.config"}
DEBUG: Available deps: []
DEBUG: Missing deps  : []
DEBUG: Predirs: []
==> rel (generate)
DEBUG: Consult config file "/home/lloyd/Erl/nitrogen/rel/reltool.config"
DEBUG: vcs_vsn: Unknown VCS atom in vsn field: "2.2.2"
DEBUG: vcs_vsn: Unknown VCS atom in vsn field: []
ERROR: generate failed while processing /home/lloyd/Erl/nitrogen/rel: {'EXIT',{{badmatch,{error,"Illegal library \"/home/lloyd/Programming/Erlang/zippity/apps\": no such file or directory"}},
         [{rebar_reltool,generate,2,[]},
          {rebar_core,run_modules,4,[]},
          {rebar_core,execute,5,[]},
          {rebar_core,process_dir1,6,[]},
          {rebar_core,process_commands,2,[]},
          {rebar,main,1,[]},
          {escript,run,2,[{file,"escript.erl"},{line,747}]},
          {escript,start,1,[{file,"escript.erl"},{line,277}]}]}}
make[2]: *** [generate] Error 1
make[2]: Leaving directory `/home/lloyd/Erl/nitrogen'
make[1]: *** [rel] Error 2
make[1]: Leaving directory `/home/lloyd/Erl/nitrogen'
make: *** [rel_inets] Error 2

A: Big thanks guys. Evidently I had set ERL_LIBS sometime back not fully understanding the implications. The ERL_LIBS value was orphaned when I deleted the library directory. So, unset ERL_LIBS did the trick. Nitrogen is now installed.
Re hosed Erlang installation: thanks to tip from Bob Ippolito, I purged Erlang from my system, installed the Erlang dependencies, then reinstalled Erlang via kerl. Rebar now compiles. n2o is still not cooperating--- stalls during compilation after many warnings. But I'll work on that today.
Thanks again,
LRP


text:
Q: Hamilton's function according to the Delaunay variables Consider the following system:
\begin{equation}
\ddot{x}_1=-\frac{\mu_{\oplus} x_1}{r^3}-\frac{\mu_\oplus R_{\oplus}^2J_2}{r^5}\bigg(\frac{3}{2}x_1-\frac{15}{2}\frac{x_1x_3^2}{r^2}\bigg)\\
\ddot{x}_2=-\frac{\mu_{\oplus} x_2}{r^3}-\frac{\mu_\oplus R_{\oplus}^2J_2}{r^5}\bigg(\frac{3}{2}x_2-\frac{15}{2}\frac{x_2x_3^2}{r^2}\bigg)\\
\ddot{x}_2=-\frac{\mu_{\oplus} x_3}{r^3}-\frac{\mu_\oplus R_{\oplus}^2J_2}{r^5}\bigg(\frac{9}{2}x_3-\frac{15}{2}\frac{x_3^3}{r^2}\bigg).
\end{equation}
where $J_2=1082.6261\cdot 10^{-6}, \mu_{\oplus}=398600.4405km^3s^{-2},r=\sqrt{x_1^2+x_2^2+x_3^2}, R_{\oplus}=6378.137 km.$
Give me an idea of how I could prove that this system, which describes the case of a satellite whose motion is disturbed by the non-sphericity of the earth, is a Hamiltonian system and how I could determine Hamilton's function in Delaunay coordinates.
I understand the practical part well, but from a mathematical point of view I do not have a theoretical basis.
Thank you very much !


text:
Q: Trigger ngInfiniteScroll correctly when using a scrolling table I have a table inside a bootstrap-modal which has overflow set to auto so that it scrollable. However I need to load a lot of data so I would like to implement lazy-loading using ngInfiniteScroll. I have tried using the infinite-scroll-container attribute but it's still triggering constantly. Here is a code-snippet:

            CODE
            TYPE
            PRODUCT
            ONBOARD

            {{sheet.code}}
            {{sheet.type}}
            {{sheet.product}}

                    <input type="checkbox" data-ng-model="sheet.onboard" data-ng-value="sheet.onboard">

                Click to select
                <us-list-select ng-model="sheet.located"
                                data-ng-show="sheet.onboard"
                                options="{list:vm.storageLocations}">
                </us-list-select>

Any ideas?


text:
Q: inline jsdoc for return I know how to inline types for parameters:
function foo(/**number*/theNum, /**string*/theText){}

But how do I describe the return type using this method?
I'm not talking about the way to do it above the function, like this:
/** this method takes too much space!
 * @returns {boolean}
*/
function foo(){ return false; }

I want to describe it inline.

A: According to this:
function /** boolean */ foo(){ return false; }


text:
Q: JaxB - how to not map a specific XML node Here is my problem:
I'm working on a project to migrate from Castor to JaxB. I am very new to JaxB and facing a problem which boggles my mind and yet, may be quite simple to solve. Here is a simple example :
class Data
{
    private Item item;
}

I have Data containing one Item. Here is how the output XML file (which I cannot change) looks like:
<Data>
    <Items>
        <Item>
        <Item>
    <Items>
<Data>

Even though Data contains only one Item, it should be presented in the output XML as an element in Items. The thing is, I do not want JaxB to generate a class or property for Items, only Data and Item.
How should my schema look like? Is there any way to specify in the schema that the specified item is stored in a node which has no class representation? Something which in my logic could look like that:
  <xs:element name="Data">
    <xs:complexType>
      <xs:sequence>
        <xs:element ref="Items"/>
      </xs:sequence>
    </xs:complexType>
  </xs:element>

  <xs:element name="Items"> <!-- Do not generate plz -->
    <xs:complexType>
      <xs:sequence>
        <xs:element ref="Item" minOccurs="1" maxOccurs="1"/>
      </xs:sequence>
    </xs:complexType>
  </xs:element>

A: Note:  I'm the EclipseLink JAXB (MOXy) lead and a member of the JAXB 2 (JSR-222) expert group.
For the following fragment, there isn't a way to generate the classes in that way, because the standard JAXB APIs don't provide path based mapping.
  <xs:element name="Items"> <!-- Do not generate plz -->
    <xs:complexType>
      <xs:sequence>
        <xs:element ref="Item" minOccurs="1" maxOccurs="1"/>
      </xs:sequence>
    </xs:complexType>
  </xs:element>

However, starting from Java classes you could use MOXy's @XmlPath extension to map this use case:
import org.eclipse.persistence.annotations.oxm.XmlPath;

class Data
{
    @XmlPath("Items/Item")
    private Item item;
}

For More Information

*

*http://blog.bdoughan.com/2010/07/xpath-based-mapping.html

*http://blog.bdoughan.com/2011/05/specifying-eclipselink-moxy-as-your.html

A: One option is to write your own jaxb plugin: you can find an introduction here: http://weblogs.java.net/blog/kohsuke/archive/2005/06/writing_a_plugi.html. Within your plugin, you generate code using the Codemodel API; I've blogged about that here: http://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/21495. When XJC gets to your "Items" element, it will pass control to you and in your plugin, you'd simply ignore it and create the setter/getter for the underlying "Item" element instead.  I'm sure there are other variations that are just as valid.

A: Using standard JAXB annotations:
class Data{
  @XmlElementWrapper(name="Items")
  @XmlElement(name="Item")
  private Item[] item;     // An array with just one Item
}


text:
Q: Image Dodging (color matching algorithm) in ERDAS I have two LISS 3 images acquired in Feb and March a year apart. I have performed atmospheric correction (haze reduction) for both the images individually before applying the color matching algorithm while mosaicing. I applied the image dodging color matching algorithm in the mosaicpro tool. The output was fine, except that the edges of the two images seemed to be lightened, i.e., the mosaic was not seamless in the edges. It could be a radiometric error for the edges of the two images. I could not find any specific parameters in the tool to overcome this error.
I have tried this procedure (color matching two images before mosaicing) in ArcGIS and ENVI but the results were not satisfactory.
Can I overcome this error using ERDAS?

A: In order to eliminate this issue you have to use seamline, Keep in mind while editing seamline try to digitized the seamline in center of common overlap area. The outer edge will be lightened.There is a way to overcome the edge lightened problem too. Approached is different need some manipulation

A: This effect is very likely due to surface bidirectional effect, which generally causes high reflection at large view angle. You can use a brdf model to correct it such as Walthall model.


text:
Q: Python regex - getting timestamp from string I am using Python 2.7
I have a string which takes form similar to these:
A: u'As of 9:30 AM EDT.
B: u'As of 10:15 AM EDT.
C: u'As of 4:30 PM EDT.

Here is my code:
import re

pat= u'[0-9]?[1-9]{1}:[0-9]{2}\s[A|P]M'
rxc = re.compile(pat)

s = grok_date_string()

matches = re.findall(rxc, s)

My code fails to match B. Why?!

A: The second digit of the hours may not be 0, only 1-9. Thus your code would be matching 11:15 AM but not 10:15 AM.
Replace the regex with
'[0-9]?[0-9]{1}:[0-9]{2}\s[A|P]M'

and it should work.
You can even simplify it further using
'[0-9]{1,2}:[0-9]{2}\s[A|P]M'

A: Here is a link to the example.
import re

pat = re.compile(r'(\d{1,2}:\d{2}\s(?:AM|PM))')
results = re.findall(pat, YOUR_STRING)
# matches HH:MM AM/PM

A: The issue is with the second digit, that is restricted by your regex to the range of 1-9, which will not match your second example.
A possible regex could be: [01]?[0-9]:[0-5][0-9]\s[AP]M
Note: you don't need to put the | character between square brackets (as you did to match AM/PM); all the characters within the square brackets are automatically OR'd.


text:
Q: Covid-19 Growth rate (Bootstrapping/Time Series) I am trying to code R in order to obtain growth rate for COVID-19.
The equation can be found on the inserted image where i(t) is the number of infected individuals at time t.
I think I could code if the equation were to be simple growth rate i(t) = i0 * exp(r*t) for example; however, the paper I am referring to implemented following method: "Time series data at each time point, a bootstrap method of extracting 1000 repeated observations was used, assuming that it follows a Poisson distribution with it^Data as the mean."
I have never coded such equations before :( I'm hoping this to be the great opportunity & initiation of being better coder & statistician.
Can somebody please help me out with this? I would really appreciate any type of hints or suggestions. I am never expecting whole code or anything near that; I would love to find a source where I could start this off well.

A: Here's an example using some AIDS case data (taken from Dobson and Barnett's GLM introduction).
aids_dat <- read.csv("https://raw.githubusercontent.com/bbolker/goettingen_2019/master/data/aids.csv")
## generate more useful time variable
aids_dat <- transform(aids_dat, time = year - min(year) + (quarter-1)/4,
                      cumcases = cumsum(cases))
nobs <- nrow(aids_dat)

log-linear fit to the data
(you would use whatever kind of model you wanted here)
f1 <- lm(log(cases) ~ time, data = aids_dat)

Generate simulated data sets
set.seed(101)
fitted <- predict(f1)
s <- replicate(1000,
               rpois(nobs, lambda = exp(fitted)),
               simplify = FALSE)

This takes the predicted values from the log-linear fit and adds Poisson variation. Chowell et al. do a bunch of stuff with the cumulative cases numbers that I don't understand (and generally think isn't a great idea anyway); it looks like they generate predicted values, create a cumulative case count, and immediately difference it to get back to cases? It's worth being careful about this stuff (keep in mind the difference between prevalence and incidence), but I'm not quite sure what Chowell et al. are doing.
refit the model
refit <- lapply(s,
                function(x) lm(log(x+0.1) ~ time, data = aids_dat))

I used log(x+0.1) in case the Poisson sample generated zero values. (It would be better to use a Poisson GLM or something.)
get predicted values for each refitted model
boot_ensemble <- sapply(refit, predict)

You could also use *apply or a for loop to extract coefficients from each refitted model, compute epidemic metrics (growth rate or R0), etc. — whatever values you want the bootstrap distribution for.
plot
png("boot.png")
plot(cases ~ time, data = aids_dat)
matlines(aids_dat$time, exp(boot_ensemble),
         type = "l", lty = 1,
         col = adjustcolor("black", alpha = 0.01))
matlines(aids_dat$time,
         col = "red", lty = 1,
         t(apply(exp(boot_ensemble), 1, range)))
dev.off()

You can get pointwise confidence intervals by applying quantile to each row of the prediction ensemble, although see Juul et al 2021 for cautionary notes. (You can use the fda::fbplot() function to plot similar curves to theirs.)
Final cautionary note, I suspect that using Poisson (rather than e.g. negative binomial) parametric bootstrapping will often underestimate uncertainty.
Juul, Jonas L., Kaare Græsbøll, Lasse Engbo Christiansen, and Sune Lehmann. "Fixed-Time Descriptive Statistics Underestimate Extremes of Epidemic Curve Ensembles." Nature Physics 17, no. 1 (January 2021): 5–8. https://doi.org/10.1038/s41567-020-01121-y.


text:
Q: No field of R can be resolved I have a little project and yesterday i tried out the clean function ("Project/clean").
But after the clean no "id" or "layout" can be resolved ("[name] cannot be resolved or is not a field").
I have no idea what I have done wrong, because i have read that this is a good way to clean up your whole project...
Anyone a idea how to resolve this?
Thanks

A: Check for errors in XML files. The R class is generated after XML is compiled to allow it to talk to Java code. If it can't find R, either the XML didn't build or couldn't build.
Also, make sure you're not importing com.android.R. Your project has its own R class: com.your.package.name.R.

A: R.java is regenerated on the next build after every clean. If you don't have "Build Automatically" enabled, try enabling it to start the regeneration right after the clean. Try running the project (or just building it). If that fails, Save the project and restart your IDE.

A: I generally fix resource errors by deleting stuff.  First, delete imports that include R.  Next, if the first step didn't help, delete the contents of /gen/ (but not the directory itself).


text:
Q: Checkbox violates data validation rules I've generated multiple checkboxes (all are unticked) using apps script, all with the same code but some of them when clicked result in a popup saying they violated data validation rules (and they remain unticked). I checked manually (in the data validation menu) and they all have the same rule and if I remove the rule nothing changes.

var enforceCheckbox = SpreadsheetApp.newDataValidation(); enforceCheckbox.requireCheckbox(); enforceCheckbox.setAllowInvalid(false); enforceCheckbox.build(); checkboxRange.setDataValidation(enforceCheckbox);
In my script all their values are checked using the onEdit event but since I can't even edit their value I doubt that's relevant.

A: Turns out the problem was that the cells affected were formatted as plain text. Turning the formatting to automatic solved the issue.
Thanks to @Cooper I've discovered the simple way to add checkboxes via code, sadly this doesn't seem to be well documented on the apps script reference page.
For other beginners stumbling on this you can add a checkbox using:
  range.insertCheckboxes();
  range.insertCheckboxes(checkedValue);
  range.insertCheckboxes(checkedValue, uncheckedValue);

You can remove the checkboxes with:
  range.removeCheckboxes();


text:
Q: How can I force Entity Framework Code First to accept "out of phase" foreign key types, db table vs .NET entity? I have an inherited Code First Entity Framework project where the entities in C# have fallen out of phase with the database. When I attempt to create a parent-to-child 1-to-1 relationship with an existing table, EF is unable to create valid SQL for establishing the foreign key (FK) relationship.
In this case, the parent entity is called ApplicantRegistration. I need to relate this, one-to-one, with the User entity to represent a CaseManager that belongs to each applicant. (The User entity is used as a child relation in many more parent entities.)
Simple enough on its face. I added the following to the ApplicantRegistration...
public User CaseManager { get; set; }

But the User has an id that says this...
    public class User
    {
        [Key]
        public int UserId { get; set; }
    // ...

... and, in MySQL, the Users table related to that entity has this...
  `UserId` bigint(15) NOT NULL AUTO_INCREMENT,

>>> Since bigint and int don't jibe, I get an error when EF tries to create a Foreign Key. <<<
The int column (EF names it CaseManager_UserId) gets added to the ApplicantRegistrations table fine, but the FK creation borks.
Here's the autogenerated EF SQL:
ALTER TABLE ApplicantRegistrations
ADD COLUMN CaseManager_UserId int NULL
CREATE INDEX IX_ApplicantRegistrations_CaseManager_UserId ON ApplicantRegistrations (CaseManager_UserId)
ALTER TABLE ApplicantRegistrations
  ADD CONSTRAINT FK_ApplicantRegistrations_Users_CaseManager_UserId FOREIGN KEY (CaseManager_UserId) REFERENCES Users (UserId)
  ON DELETE NO ACTION ON UPDATE NO ACTION

ADD CONSTRAINT FK_ApplicantRegistrations_Users_CaseManager_UserId FOREIGN KEY (CaseManager_UserId) isn't going to work, because int and bigint don't match.

Options Considered
1. Swap int to long in User in the C#
I originally started swapping from int for UserId on the User entity over to long (this is in C#) to see if that'd match up with bigint in EF's generated code, but that quickly became a mess, practically speaking. After I swapped public int UserId { get; set; } over to public long UserId { get; set; } and fixed the fallout from changing type on related entities & code, I received a number of errors like the following when trying to Add-Migration:

System.Data.Entity.ModelConfiguration.ModelValidationException: One or more validation errors were detected during model generation:
ProgramWorksite_WorksiteAdministratorUser_Target_ProgramWorksite_WorksiteAdministratorUser_Source: : The types of all properties in the Dependent Role of a referential constraint must be the same as the corresponding property types in the Principal Role. The type of property 'WorksiteAdministratorUserId' on entity 'ProgramWorksite' does not match the type of property 'UserId' on entity 'User' in the referential constraint 'ProgramWorksite_WorksiteAdministratorUser'.

That makes sense, of course -- now I've got to change all the other places User is used as a child relation. (There are 33 of them, and it occurs to me there should be an easier way, so I shelved int to long in User for now in the hopes of finding something better, but I'll probably come back to this after posting.)
2. Swap bigint for int in Users in MySQL
I also tried to swap from bigint to int in MySQL, but that would require dropping & replacing tons of FK relationships with UserId in the database, which is a possible solution, but not much fun. I think fixing it in the C# would be easier -- assuming long will map to bigint, which I don't know for sure. (UPDATE: long does map to bigint(15), at least in this system.)
3. Try annotating the column in ApplicantRegistration
I thought I might be able to use a Column override for EF's purposes only like this inside of Applicant...
Column("CaseManagerId", TypeName = "bigint(15)")]
public User CaseManager { get; set; }

Unfortunately, that gave the same error as before -- and it didn't use the CaseManagerId name I'd used, instead defaulting back to CaseManager_UserId. So, at best, I've done the annotation wrong.
The annotation route really is my favorite, if there's a way to make it work -- if it could make EF ignore what's in the User entity's UserId definition long enough to create the FK relationship and complete the migration, I'm golden.

What's the most efficient way to insert this new relationship if I don't want to drop EF Code First?

A: Okay, I'm hoping someone will come back and tell me there's a better way and I can move the check, but for now, here's what I'm doing...

Since all Add-Migration is really doing is auto-generating some C#, you can take that autogenerated code and hack it to match your use case -- in this case, ignoring that the entity is using int and the database is using bigint/long.
Hacking the migration code before running Update-Database breaks nothing. That is, there's nothing checking to make sure your migrations for your datastore match your entity definitions (in code) before the migration C# is run.
In my case, that means I needed to change this Up method from what the autogenerated code was...
        public override void Up()
        {
            AddColumn("dbo.ApplicantRegistrations",
                "CaseManager_UserId", c => c.Int());

            CreateIndex("dbo.ApplicantRegistrations", "CaseManager_UserId");
            AddForeignKey("dbo.ApplicantRegistrations", "CaseManager_UserId", "dbo.Users", "UserId");
        }

... to this...
        public override void Up()
        {
            AddColumn("dbo.ApplicantRegistrations",
                "CaseManager_UserId", c => c.Long()); // <<< This from Int to Long

            CreateIndex("dbo.ApplicantRegistrations", "CaseManager_UserId");
            AddForeignKey("dbo.ApplicantRegistrations", "CaseManager_UserId", "dbo.Users", "UserId");
        }

This is smelly.
As @Ctznkane525 remarks, since we're unlikely to get past 2147483647 users in this application, bigint is overkill, and the maintainers of this system (possibly including me) should eventually take option #2 from the OP, and fix the danged schema at some point.

So the quick fix (and the answer to this specific question) is to make EF ignore the out of phase entity and schema by hacking the migration file/C# code.

The Long Term Right Thing To Do, however, is to make the database schema and entity code match. Maybe you bcp the data out, massage the db with int from bigint, and recreate the schema if production can stand some downtime, or maybe you bite the bullet and stick with insanely big bigints in your schema and finish doing #1 from the question to make the EF code match and rerun your initial migration. But for now, a hack it is.


text:
Q: Why $\frac{a^{2m}-1}{a+1}=\frac{a^2-1}{a+1}(a^{2m-2}+a^{2m-4}+\cdots+a^2+1)?$ I know that $\dfrac{a^2-b^2}{a+b} = a-b$, because$$
a^2-b^2 = aa -ab+ab- bb = a(a-b)+(a-b)b = (a-b)(a+b).
$$
Also, I know that$$
\frac{a^n-b^n}{a+b} = a^{n-1}-a^{n-2}b+\cdots-ab^{n-2}+ b^{n-1}.$$
But I do not understand this equality below:
$$\frac{a^{2m}-1}{a+1}=\frac{a^2-1}{a+1}(a^{2m-2}+a^{2m-4}+\cdots+a^2+1).$$

A: $(a^2-1)(a^{2m-2}+a^{2m-4}+\cdots+a^2+1)$
$=a^{2m}+a^{2m-2}+a^{2m-4}+\cdots+a^4+a^2-(a^{2m-2}+a^{2m-4}+\cdots+a^2+1)$
$=a^{2m}+a^{2m-2}-a^{2m-2}+a^{2m-4}-a^{2m-4}+\cdots+a^4-a^4+a^2-a^2-1$
$=a^{2m}-1$
$\Rightarrow \dfrac{a^{2m}-1}{a+1}=\dfrac{a^2-1}{a+1}(a^{2m-2}+a^{2m-4}+\cdots+a^2+1)$

A: Note that, $(a^2-1)(a^{2m-2}+a^{2m-4}+\dots+a^2+1)\\=a^2(a^{2m-2}+a^{2m-4}+\dots+a^2+1)-(a^{2m-2}+a^{2m-4}+\dots+a^2+1)\\=a^{2m}+a^{2m-2}+a^{2m-4}+\dots+a^2-(a^{2m-2}+a^{2m-4}+\dots+a^2+1)\\=a^{2m}-1$

A: Because$$
a^{2m} - 1 = (a^2)^m - 1 = (a^2 - 1)((a^2)^{m - 1} + (a^2)^{m - 2} + \cdots + 1).
$$


text:
Q: jQuery trigger an event while form is posting Was just wondering if there is a simple way to trigger an event (such as displaying a loading spinner) once the form action is started (form starts posting) until the action is done (then the spinner disappear). Something like displaying spinner when ajaxStart and hiding it when ajaxStop posting.

A: This is really easy to do if the form is posted via ajax:
$("#throbber").show();
$("form").ajaxSubmit().done(function () {
   $("#throbber").hide();
});

Just put the throbber <img> in a sensible place in the html and set it to display: none initially.

A: Assuming you mean regular HTML form submission, then yes, you can intercept the submission with a form.onsubmit = function(){...} handler.
Return false to inhibit submission, or true (or anything other than false), to allow it.
In jQuery :
$("#myForm").on('submit', function(){
    ...
    return true;//or false
});

However, depending on how fast the server responds, the spinner may never show or only for a very brief moment, before the page is renewed.


text:
Q: Expanding a list based on a number per line So I have a file similar to this
A 1 foo bar 1
A 1 foo bar 2
B 3 foo bar 5
B 3 foo bar 6
Where the first column is an ID, the second column is the number of times I need the line repeated. The output should look like this
A foo bar 1
A foo bar 2
B foo bar 5
B foo bar 6
C foo bar 5
C foo bar 6
D foo bar 5
D foo bar 6
However, since the files I am trying to use are rather big, the expanding is ultra slow. Any ideas? :)
EDIT1: Code I did, I did have two inputs, one file with the columns i showed before and one with just IDs and counts separated by a dash.
for ID in $(cat $ID_File);do
 grep "^$ID[^0-9]" $IN > temp2.txt
 Cycles=$(echo $ID| sed 's/.*-//')
 for i in $(seq 1 1 $Cycles);do
  cut -f2- temp2.txt| awk '{print '$Start_ID'"\t"$0}'>> $OUT
  Start_ID=$(( $Start_ID + 1 ))
 done
done

A: I found the answer! Instead of looking through the file several times with grep I used the following Python script that goes through the file just once.
FILE='A.txt'
ID=''
Prefix='A'

Number=1
First=1
Current=[]
OUT='B.txt'
f=open(File,'w')

for line in open(FILE):
 line1=line.split('\t')
 if ID!=line1[0] and First==0:
  ID=line1[0]
  for i in range(0,int(Current[0][1])):
   for j in range(0,len(Current)):
    f.write(' '.join([Prefix+str(Number)]+Current[j][2:]))
  Number=Number+1
  Current=[]
  Current.append(line1)
 elif ID!=line1[0] and First==1:
  ID=line1[0]
  First=0
  Current.append(line1)
 elif ID==line1[0]:
  Current.append(line1)

Not the most elegant way to do it, but it gets the job done fast relatively quickly.


text:
Q: How to retain drawn boxes and path on page refresh mxgraph i want to retain drawn boxes and path once page is refreshed.
i'm using mxgraph https://jgraph.github.io/mxgraph/docs/manual.html
Question:  once the drawing is done, if i do page refresh it should retain the same drawing by storing drawing data into localStorage.
NOTE: i want to get json object of drawn canvas, and re-construct it back from that object on page refresh.
Below video shows how to draw: https://drive.google.com/file/d/1McTMz3e8I_quOcLGt2CtaQhqif3R31qs/view

https://jgraph.github.io/mxgraph/javascript/examples/editors/images/
<!--
  Copyright (c) 2006-2013, JGraph Ltd

  Dynamic toolbar example for mxGraph. This example demonstrates changing the
  state of the toolbar at runtime.
-->
<html>
<head>
 <title>Toolbar example for mxGraph</title>

 <!-- Sets the basepath for the library if not in same directory -->
  <script type="text/javascript">
    mxBasePath = 'https://jgraph.github.io/mxgraph/javascript/src';
  </script>

  <!-- Loads and initializes the library -->
  <script type="text/javascript" src="https://jgraph.github.io/mxgraph/javascript/src/js/mxClient.js"></script>

 <!-- Example code -->
 <script type="text/javascript">
  // Program starts here. Creates a sample graph in the
  // DOM node with the specified ID. This function is invoked
  // from the onLoad event handler of the document (see below).
  function main()
  {
   // Checks if browser is supported
   if (!mxClient.isBrowserSupported())
   {
    // Displays an error message if the browser is
    // not supported.
    mxUtils.error('Browser is not supported!', 200, false);
   }
   else
   {
    // Defines an icon for creating new connections in the connection handler.
    // This will automatically disable the highlighting of the source vertex.
    mxConnectionHandler.prototype.connectImage = new mxImage('images/connector.gif', 16, 16);

    // Creates the div for the toolbar
    var tbContainer = document.createElement('div');
    tbContainer.style.position = 'absolute';
    tbContainer.style.overflow = 'hidden';
    tbContainer.style.padding = '2px';
    tbContainer.style.left = '0px';
    tbContainer.style.top = '0px';
    tbContainer.style.width = '24px';
    tbContainer.style.bottom = '0px';

    document.body.appendChild(tbContainer);

    // Creates new toolbar without event processing
    var toolbar = new mxToolbar(tbContainer);
    toolbar.enabled = false

    // Creates the div for the graph
    var container = document.createElement('div');
    container.style.position = 'absolute';
    container.style.overflow = 'hidden';
    container.style.left = '24px';
    container.style.top = '0px';
    container.style.right = '0px';
    container.style.bottom = '0px';
    container.style.background = 'url("editors/images/grid.gif")';

    document.body.appendChild(container);

    // Workaround for Internet Explorer ignoring certain styles
    if (mxClient.IS_QUIRKS)
    {
     document.body.style.overflow = 'hidden';
     new mxDivResizer(tbContainer);
     new mxDivResizer(container);
    }

    // Creates the model and the graph inside the container
    // using the fastest rendering available on the browser
    var model = new mxGraphModel();
    var graph = new mxGraph(container, model);

    // Enables new connections in the graph
    graph.setConnectable(true);
    graph.setMultigraph(false);

    // Stops editing on enter or escape keypress
    var keyHandler = new mxKeyHandler(graph);
    var rubberband = new mxRubberband(graph);

    var addVertex = function(icon, w, h, style)
    {
     var vertex = new mxCell(null, new mxGeometry(0, 0, w, h), style);
     vertex.setVertex(true);

     var img = addToolbarItem(graph, toolbar, vertex, icon);
     img.enabled = true;

     graph.getSelectionModel().addListener(mxEvent.CHANGE, function()
     {
      var tmp = graph.isSelectionEmpty();
      mxUtils.setOpacity(img, (tmp) ? 100 : 20);
      img.enabled = tmp;
     });
    };

    addVertex('https://jgraph.github.io/mxgraph/javascript/examples/editors/images/rectangle.gif', 100, 40, '');
    addVertex('https://jgraph.github.io/mxgraph/javascript/examples/editors/images/rounded.gif', 100, 40, 'shape=rounded');
    addVertex('https://jgraph.github.io/mxgraph/javascript/examples/editors/images/ellipse.gif', 40, 40, 'shape=ellipse');
    addVertex('https://jgraph.github.io/mxgraph/javascript/examples/editors/images/rhombus.gif', 40, 40, 'shape=rhombus');
    addVertex('https://jgraph.github.io/mxgraph/javascript/examples/editors/images/triangle.gif', 40, 40, 'shape=triangle');
    addVertex('https://jgraph.github.io/mxgraph/javascript/examples/editors/images/cylinder.gif', 40, 40, 'shape=cylinder');
    addVertex('https://jgraph.github.io/mxgraph/javascript/examples/editors/images/actor.gif', 30, 40, 'shape=actor');
   }
  }

  function addToolbarItem(graph, toolbar, prototype, image)
  {
   // Function that is executed when the image is dropped on
   // the graph. The cell argument points to the cell under
   // the mousepointer if there is one.
   var funct = function(graph, evt, cell, x, y)
   {
    graph.stopEditing(false);

    var vertex = graph.getModel().cloneCell(prototype);
    vertex.geometry.x = x;
    vertex.geometry.y = y;

    graph.addCell(vertex);
    graph.setSelectionCell(vertex);
   }

   // Creates the image which is used as the drag icon (preview)
   var img = toolbar.addMode(null, image, function(evt, cell)
   {
    var pt = this.graph.getPointForEvent(evt);
    funct(graph, evt, cell, pt.x, pt.y);
   });

   // Disables dragging if element is disabled. This is a workaround
   // for wrong event order in IE. Following is a dummy listener that
   // is invoked as the last listener in IE.
   mxEvent.addListener(img, 'mousedown', function(evt)
   {
    // do nothing
   });

   // This listener is always called first before any other listener
   // in all browsers.
   mxEvent.addListener(img, 'mousedown', function(evt)
   {
    if (img.enabled == false)
    {
     mxEvent.consume(evt);
    }
   });

   mxUtils.makeDraggable(img, graph, funct);

   return img;
  }

 </script>
</head>

<!-- Calls the main function after the page has loaded. Container is dynamically created. -->
<body onload="main();" >
</body>
</html>

A: This uses the XML encoding of mxgraph. You see this code in one of the examples of mxGraph. While it is not json, it is very easy to turn into json, and can be edited or saved as you wish. Now it it stored in localStorage as you asked.
Add the following lines at the end of your main function (after the last call to addVertex):
// read state on load
if(window.localStorage.graphState){
    var doc = mxUtils.parseXml(window.localStorage.graphState);
    var dec = new mxCodec(doc);
    dec.decode(doc.documentElement, graph.getModel());
}

// save state on change
graph.getModel().addListener('change',function(){
    var codec = new mxCodec();
    window.localStorage.graphState = codec.encode(
            graph.getModel()
        ).outerHTML;
});

Update using JSON
Update 2: Switched to my own custom html2json and json2html implementation to make the answer fully self-contained
If you do need a proper JSON object, you can convert from xml to json and back. Below you can see I stringify and parse the json object, this is only needed to save it in localStorage though.
First add these two functions to convert from html/xml to json and from json back to html/xml:
function html2json(html){
    if(html.nodeType==3){
        return {
            "tagName":"#text",
            "content":html.textContent
        }
    }
    var element = {
        "tagName":html.tagName
    };

    if(html.getAttributeNames().length>0){
        element.attributes = html.getAttributeNames().reduce(
            function(acc,at){acc[at]=html.getAttribute(at); return acc;},
            {}
        );
    }

    if(html.childNodes.length>0){
        element.children = Array.from(html.childNodes)
            .filter(
                function(el){
                    return el.nodeType!=3
                    ||el.textContent.trim().length>0
                })
            .map(function(el){return html2json(el);});
    }
    return element;
}

function json2html(json){
    var xmlDoc = document.implementation.createDocument(null, json.tagName);

    var addAttributes = function(jsonNode, node){
        if(jsonNode.attributes){
            Object.keys(jsonNode.attributes).map(
                function(name){
                    node.setAttribute(name,jsonNode.attributes[name]);
                }
            );
        }
    }

    var addChildren = function(jsonNode,node){
        if(jsonNode.children){
            jsonNode.children.map(
                function(jsonChildNode){
                    json2htmlNode(jsonChildNode,node);
                }
            );
        }
    }

    var json2htmlNode = function(jsonNode,parent){
        if(jsonNode.tagName=="#text"){
            return xmlDoc.createTextNode(jsonNode.content);
        }

        var node = xmlDoc.createElement(jsonNode.tagName);

        addAttributes(jsonNode,node);
        addChildren(jsonNode,node);

        parent.appendChild(node);
    }

    addAttributes(json,xmlDoc.firstElementChild);
    addChildren(json,xmlDoc.firstElementChild);

    return xmlDoc;
}

Then change my JavaScript snippet from above like this to transform the xml into json and back:
// read state on load
if(window.localStorage.graphState){
    var doc = json2html(JSON.parse(localStorage.graphState));
  var dec = new mxCodec(doc);
  dec.decode(doc.documentElement, graph.getModel());
}

// save state on change
graph.getModel().addListener('change',function(){
        var codec = new mxCodec();
    window.localStorage.graphState = JSON.stringify(html2json(codec.encode(
      graph.getModel()
    )));
});


text:
Q: List Field Name Character Limit I'm wanting to give a field a custom name that links it to a row in a separate list. The issue is the fields need to be linked to a string that is a full sentence. One solution would be to have the full sentence as the field name, but this of course wont work if the sentence is longer than the character limit. What is the character limit of a Sharepoint List Field?
Edit:
Sorry, should've been more clear. I was referring to the TITLE of the field not the actual data stored in that column. A comment to one of the answers answered my question, it is 128 characters. Thank you.

A: Document libraries and lists have different options available with respect to the multiple lines of text column type.
In document libraries, when you create a new column of type Multiple Lines of Text, you get the option to select the Allow unlimited length in document libraries checkbox.
But, In lists, this option does not exist.
A multi line text field is defined in SharePoint as a Note type, this is stored in the DB as a ntext sql type. In this SQL type you can store 2gb of char data, that's 1,073,741,823 characters.
Note: It is not advisable to store such huge data in SharePoint column as it may hamper performance.
Explanation referred from this
    https://social.technet.microsoft.com/Forums/sharepoint/en-US/5e8cc504-03d0-4d26-81f9-1608c2c5881f/what-is-the-max-limit-for-number-of-characters-in-multiline-text-in-sharepoint?forum=sharepointgenerallegacy
I hope this is a helpful information!!

A: The character limit would be 255 characters for Name / Title. But it is recommended to have Name shorter as the Name is going to be used in queries if required.
Let me know if my assumption is wrong.


text:
Q: Boost ASIO, async_read_some callback not called My code works for read_some, but not for async_read_some.  The data I'm reading is 5 chars long, whereas MAX_RESPONSE_SIZE 256.  I call async_read_some once from my main after opening the port, but the callback is never called after I swipe my prox card a few times.  I have tried adding io_service.run() after async_read_some but it did not help.  Am I missing something?  Thank you.
header
boost::system::error_code error;
boost::asio::io_service io_service;
typedef boost::shared_ptr<boost::asio::serial_port> serial_port_ptr;
serial_port_ptr serial_port;
char read_buffer[MAX_RESPONSE_SIZE];

open
serial_port.reset();
serial_port = serial_port_ptr(new boost::asio::serial_port(io_service));
serial_port->open(device_path, error);

serial_port->set_option(boost::asio::serial_port_base::baud_rate(baud_rate));
serial_port->set_option(boost::asio::serial_port_base::character_size(8));
serial_port->set_option(boost::asio::serial_port_base::stop_bits(boost::asio::serial_port_base::stop_bits::one));
serial_port->set_option(boost::asio::serial_port_base::parity(boost::asio::serial_port_base::parity::none));
serial_port->set_option(boost::asio::serial_port_base::flow_control(boost::asio::serial_port_base::flow_control::none));

boost::thread t(boost::bind(&boost::asio::io_service::run, &io_service));

read
serial_port->async_read_some(
            boost::asio::buffer(read_buffer, MAX_RESPONSE_SIZE),
            boost::bind(
                &serial_comm::data_received,
                this, boost::asio::placeholders::error,
                boost::asio::placeholders::bytes_transferred
            )
        );

callback
void serial_comm::data_received(const boost::system::error_code& error, size_t bytes_transferred)
{
    // do stuff
}

A: You must ensure that there is always work to do, so that io_service::run() does not return and complete the thread where it is running.
As mentioned in the comments, you can create an io_service::work.  However, I consider this artificial, a symptom of a design problem.
The better answer, probably, is that in the data_received handler, you should prepare for the next read if no fatal error occurred
void serial_comm::data_received(
  const boost::system::error_code& error,
  size_t bytes_transferred)
{
    // do stuff

   if( any_kind_of_fatal_error )
   {
       // return without setting up the next read
       // this will end reading
      return;
   }

   // the last read was successful
   // so setup for the next
   serial_port->async_read_some(
        boost::asio::buffer(read_buffer, MAX_RESPONSE_SIZE),
        boost::bind(
            &serial_comm::data_received,
            this, boost::asio::placeholders::error,
            boost::asio::placeholders::bytes_transferred
        )
    );

}

A: Basically my problem was not starting the io_service thread after async_read_some in the same function.  Can you blame me?  This stuff is not very clear cut.  Here's my code in case anyone wants it (INFO and ERROR come from boost logging, see one of my other questions on it):
serial_comm.hpp
#ifndef __SERIAL_COMM_HPP
#define __SERIAL_COMM_HPP

#include <boost/asio.hpp>
#include <boost/asio/serial_port.hpp>
#include <boost/system/error_code.hpp>
#include <boost/system/system_error.hpp>
#include <boost/bind.hpp>
#include <boost/thread.hpp>

#include <string>
#include <atomic>

#include "logging.hpp"  // Boost logging

#define MAX_RESPONSE_SIZE 256

class serial_comm
{
    public:
        void open_serial_port (std::string device_path, unsigned int baud_rate);
        void close_serial_port (void);
        void async_read_some (void);

        std::string serial_read_data;
        std::atomic <bool> serial_data_read_complete{false};
    private:
        // functions
        void data_received (const boost::system::error_code& ec, size_t bytes_transferred);

        // variables
        boost::mutex mutex;
        boost::system::error_code error;
        boost::asio::io_service io_service;
        typedef boost::shared_ptr<boost::asio::serial_port> serial_port_ptr;
        serial_port_ptr serial_port;
        char read_buffer[MAX_RESPONSE_SIZE];
};

#endif  // __SERIAL_COMM_HPP

serial_comm.cpp
#include "../include/serial_comm.hpp"

void serial_comm::open_serial_port (std::string device_path, unsigned int baud_rate)
{
    INFO << "started";

    try
    {
        serial_port.reset();
        serial_port = serial_port_ptr(new boost::asio::serial_port(io_service));
        serial_port->open(device_path, error);
        if (error)
        {
            ERROR << "error.message() >> " << error.message().c_str();
            throw -3;
        }
        // set options
        serial_port->set_option(boost::asio::serial_port_base::baud_rate(baud_rate));
        serial_port->set_option(boost::asio::serial_port_base::character_size(8));
        serial_port->set_option(boost::asio::serial_port_base::stop_bits(boost::asio::serial_port_base::stop_bits::one));
        serial_port->set_option(boost::asio::serial_port_base::parity(boost::asio::serial_port_base::parity::none));
        serial_port->set_option(boost::asio::serial_port_base::flow_control(boost::asio::serial_port_base::flow_control::none));

    }
    catch (int error)
    {
        ERROR << "error = " << error;
        throw -1;
    }
    catch (const std::exception &e)
    {
        ERROR << "e.what() = " << e.what();
        throw -2;
    }

    INFO << device_path << " opened correctly";
    INFO << "ended";

    return;
}

void serial_comm::close_serial_port()
{
    boost::mutex::scoped_lock lock(mutex);  // prevent multiple thread access

    INFO << "started";

    try
    {
        if (serial_port)
        {
            serial_port->cancel();
            serial_port->close();
            serial_port.reset();
        }
        else
        {
            WARNING << "serial port is not open";
        }
        io_service.stop();
        io_service.reset();
    }
    catch (const std::exception &e)
    {
        ERROR << "e.what() = " << e.what();
        throw -1;
    }

    INFO << "ended";

    return;
}

void serial_comm::async_read_some (void)
{
    boost::mutex::scoped_lock lock (mutex); // prevent multiple threads

    INFO << "started";

    std::string data;

    try
    {
        if (serial_port.get() == NULL || !serial_port->is_open())
        {
            WARNING << "serial port is not open";
            throw -2;
        }

        serial_port->async_read_some(
            boost::asio::buffer(read_buffer, MAX_RESPONSE_SIZE),
            boost::bind(
                &serial_comm::data_received,
                this, boost::asio::placeholders::error,
                boost::asio::placeholders::bytes_transferred
            )
        );
        // start io_service run thread after giving it work
        boost::thread t(boost::bind(&boost::asio::io_service::run, &io_service));
    }
    catch (const std::exception &e)
    {
        ERROR << "e.what() = " << e.what();
        throw -1;
    }

    INFO << "ended";

    return;
}

void serial_comm::data_received(const boost::system::error_code& error, size_t bytes_transferred)
{
    boost::mutex::scoped_lock lock(mutex);  // prevent multiple thread access

    INFO << "started";

    try
    {
        if (serial_port.get() == NULL || !serial_port->is_open())
        {
            WARNING << "serial port is not open";
            throw -2;
        }
        if (error)
        {
            ERROR << "error.message() >> " << error.message().c_str();
            throw -3;
        }
        for (unsigned int i = 0; i < bytes_transferred; ++i) {
            serial_read_data += read_buffer[i];
        }
        INFO << "bytes_transferred = " << bytes_transferred << "; serial_read_data = " << serial_read_data;
        serial_data_read_complete = true;
    }
    catch (const std::exception &e)
    {
        ERROR << "e.what() = " << e.what();
        throw -1;
    }

    // prevent io_service from returning due to lack of work
    serial_port->async_read_some(
        boost::asio::buffer(read_buffer, MAX_RESPONSE_SIZE),
        boost::bind(
            &serial_comm::data_received,
            this, boost::asio::placeholders::error,
            boost::asio::placeholders::bytes_transferred
        )
    );

    INFO << "ended";

    return;
}

main.cpp
#include "../include/serial_comm.hpp"

int main(void)
{
    serial_comm _serial_comm;

    try
    {
        _serial_comm.open_serial_port("/dev/ttyS0", 9600);
        _serial_comm.async_read_some(); // this function will always check for data
        loop:
        while (!_serial_comm.serial_data_read_complete)
        {
            sleep(1);
        }
        INFO << "_serial_comm.serial_read_data = " << _serial_comm.serial_read_data;
        _serial_comm.serial_read_data.clear();
        _serial_comm.serial_data_read_complete = false;
        goto loop;
    }
    catch (int error)
    {
        ERROR << "error >> " << error;
        return;
    }

    FATAL << "main ended";

    return;
}


text:
Q: Copy data from a longer array to a shorter array I would like to copy only certain data from an old array of chars to a new array of chars. This is what I have thus far:
char[] charsInString = s.toCharArray();

int length = 0;
for (int i = 0; i < charsInString.length; i++) {
    if (!(charsInString[i] < 65 || charsInString[i] > 122))
        length++;
}

char[] newCharList = new char[length];
for (int i = 0; i < charsInString.length; i++) {
    // not sure what to do here?
}

I only want chars in the new array that correspond with letters in the alphabet (a, b, c, etc.), essentially copying the old char array without the chars that correspond to numbers, punctuation, spaces, etc. Is there any way to do this? I have tried using both for loops and while loops, but it is just not working. Suggestions?

A: Strip all non-alphabetic characters from the Original string before converting to a character array.
String stripped = s.replaceAll("[^a-z]", "");
char[] charsInString = stripped.toCharArray();

This solution is not the most efficient, however, unless your input String is very long this should be negligible.

A: Try this code
    String str = " @#$%@##$%$& @#$%#$   alph #$%a#$%# be&*%#@ts";
    char[] charsInString = str.toCharArray();

    StringBuilder sb = new StringBuilder();
    for (int i = 0; i < charsInString.length; i++) {
        if ((charsInString[i] > 65 && charsInString[i] < 122))
            sb.append(charsInString[i]);
    }

    char[] newCharList = sb.toString().toCharArray();

    System.out.println(newCharList);

Output:
alphabets