text:
Q: Connecting two tables so that results are shown in multiple columns Let's assume we have table with data like this
+----------------+------------+-------+
| Company_number |    Date    | Value |
+----------------+------------+-------+
|            123 | 2017-01-01 |     5 |
|            123 | 2017-02-01 |    10 |
|            123 | 2018-01-01 |    15 |
|            456 | 2018-01-05 |    33 |
+----------------+------------+-------+

What should I do to receive data in format
+----------------+------+------------+------------+
| Company_number | Mont | Value 2017 | Value 2018 |
+----------------+------+------------+------------+
|            123 |   01 |          5 |         15 |
|            123 |   02 |         10 |            |
|            456 |   01 |         33 |            |
+----------------+------+------------+------------+

I have no idea how to select all data from 2017 and connect it using the company number and month with data from 2018.
I have taken all of the records from 2017, put it into another table, and tried to use this select, but it doesn't show records when there are no common months (there is no record for February).
select
s.company_number
,datepart(month,s.date) as Month
,s.Value as Value_2017
,r.Value as Value_2018
from table1 as s
left join table2 as r on concat(r.company_number,datepart(month,r.date))=concat(s.company_number,datepart(month,s.date))
where datepart(year,s.date)='2018'

select results (with no February)
    +----------------+-------+------------+------------+
    | company_number | Month | Value_2017 | Value_2018 |
    +----------------+-------+------------+------------+
    |            123 |     1 |         15 | 5          |
    |            456 |     1 |         33 | NULL       |
    +----------------+-------+------------+------------+

A: Try this out:
SELECT
    COALESCE([t1].[company_number], [t2].[company_number]) AS [Company_Number],
    MONTH(COALESCE([t1].[date], [t2].[date])) AS [Month],
    [t1].[value] AS [2018 Value],
    [t2].[value] AS [2017 Value]
FROM
    [table1] AS [t1]
FULL OUTER JOIN
    [table2] as [t2] on [t1].[company_number] = [t2].[company_number]
    AND MONTH([t1].[date]) = MONTH ([t2].[date])

Just tried it with (can be copy/pasted in editor and executed) and it works fine:
DECLARE @t1 TABLE (Company_number INT, [Date] Date, Value INT)
DECLARE @t2 TABLE (Company_number INT, [Date] Date, Value INT)
INSERT INTO @t1 VALUES (123, '2017-01-01', 5), (123, '2017-02-01', 10)
INSERT INTO @t2 VALUES (123, '2018-01-01', 15), (456, '2018-01-05', 33)

SELECT
    COALESCE([t1].Company_number, [t2].Company_number) AS [Company_Number],
    MONTH(COALESCE([t1].[date], [t2].[date])) AS [Month],
 [t1].[value] AS [2018 Value],
    [t2].[value] AS [2017 Value]
FROM
    @t1 AS [t1]
FULL OUTER JOIN
    @t2 as [t2] on [t1].[company_number] = [t2].[company_number]
    AND MONTH([t1].[date]) = MONTH ([t2].[date])

A: This is using CTE on the same table1 NOT TWO TABLES for 2017 and 2018.
With tmp as (Select company_id,

Datepart(year, dte) as year,
Datepart(month, dte) as month,
Sum(value) as value
from tbl
Group by company_id,
Datepart(year, dte),
Datepart(month, dte))
Select coalesce(m.company_id,n.company_id) as company_id,
coalesce(m.month,n.month) as month,
m.value as value_2017,
n.value as value_2018
From (select * from tmp
Where year=2017) m
Full outer join (select * from tmp
Where year=2018) n
On m.company_id=n.company_id
and m.month=n.month
Result:
   company_number   month   Value_2017  Value_2018
    123             1        5                    5
    123             2       10          (null)
    456             1    (null)             23


text:
Q: Can't edit a document launched from a view in a dialog control in xpages I have an XPages view with the first column as a link to launch the document for that row.  I'm launching it in a dialog control and it works just fine, but it won't let me edit the document.  I have an edit button on the dialog control to put it in edit mode, but it doesn't seem to work.
There are two datasources, one for the view and one for the document.
XPage:
<?xml version="1.0" encoding="UTF-8"?>

<xc:NewMainForm><xp:this.facets><xc:NewLogin xp:key="facet_Login "></xc:NewLogin>
<xc:FooterContent xp:key="facet_Footer"></xc:FooterContent>
<xc:HomeBodyContent xp:key="facet_Body"></xc:HomeBodyContent>
<xc:MenuContainer xp:key="facet_Menu"></xc:MenuContainer>
<xc:cc_headerRightContent xp:key="facet_HeaderRight">
</xc:cc_headerRightContent>
<xc:cc_headerCenterContent xp:key="facet_HeaderCenter">
</xc:cc_headerCenterContent>
<xc:cc_headerLeftContent xp:key="facet_HeaderLeft">
</xc:cc_headerLeftContent>
<xc:NewLogin xp:key="facet_Login"></xc:NewLogin></xp:this.facets>
</xc:NewMainForm></xp:view>

Custom Control:
<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="http://www.ibm.com/xsp/core"
xmlns:xe="http://www.ibm.com/xsp/coreex"
style="background-repeat:no-repeat;background-position:center center"
xmlns:xc="http://www.ibm.com/xsp/custom">

<xp:this.resources>
    <xp:script src="/xspDialog.js" clientSide="true"></xp:script>
    <xp:styleSheet href="/Style.css"></xp:styleSheet>
</xp:this.resources>
<xp:panel>
    <xp:inputText id="Search" value="#{viewScope.searchString}">
</xp:inputText>
    <xp:button value="Search" id="button1"
        style="background-color:rgb(57,88,137);color:rgb(255,255,255)">
        <xp:eventHandler event="onclick" submit="true"
            refreshMode="partial" refreshId="viewPanel1">
        </xp:eventHandler>
    </xp:button>
</xp:panel>

<!--    VIEW PANEL -->

<xp:viewPanel id="viewPanel1" pageName="/ByAcronym.xsp"
    viewStyle="width:100%;background-
image:url(Music1+Big+twice+Faded.png);background-repeat:no-
repeat;background-position:center center;border-width:thick"
    var="thisentry" rows="27">

    <xp:this.facets>
        <xp:pager partialRefresh="true" layout="Previous Group Next"
            xp:key="headerPager" id="pager1">
        </xp:pager>
        <xp:viewTitle xp:key="viewTitle" id="viewTitle1"
            value="By Title" style="color:rgb(255,128,0);font-weight:bold">
        </xp:viewTitle>

        <xp:pager partialRefresh="true" layout="Previous Group Next"
            xp:key="footerPager" id="pager2">
        </xp:pager>
    </xp:this.facets>

    <xp:this.data>
        <xp:dominoView viewName="Songs" var="dsSongs"
            ignoreRequestParams="true">
            <xp:this.search><![CDATA[#{javascript:return
 viewScope.get("searchString")}]]></xp:this.search>
        </xp:dominoView>
    </xp:this.data>

    <!-- VIEWCOLUMN1 -->

    <xp:viewColumn columnName="Title" id="viewColumn1"
        displayAs="link" style="width: 750px">
        <xp:eventHandler event="onclick" submit="true"
            refreshMode="partial" refreshId="dialog1" id="eventHandler1">
            <xp:this.action>
                <![CDATA[#{javascript:viewScope.UNID =
 thisentry.getDocument().getUniversalID();
 getComponent('dialog1').show()}]]>
            </xp:this.action>
        </xp:eventHandler>

        <xp:viewColumnHeader value="Title" id="viewColumnHeader1">
        </xp:viewColumnHeader>
    </xp:viewColumn>
    <xp:viewColumn columnName="MusicType" id="viewColumn2">
        <xp:viewColumnHeader value="Music  Type"
            style="width: 250px" id="viewColumnHeader2">
        </xp:viewColumnHeader>
    </xp:viewColumn>
    <xp:viewColumn columnName="ComposedBy" id="viewColumn3">
        <xp:viewColumnHeader value="Composer"
            id="viewColumnHeader3">
        </xp:viewColumnHeader>
    </xp:viewColumn>
    <xp:viewColumn columnName="ArrangedBy"
        id="viewColumn4">
        <xp:viewColumnHeader value="Arranger"
            id="viewColumnHeader4">
        </xp:viewColumnHeader>
    </xp:viewColumn>
    <xp:viewColumn columnName="Jonra" id="viewColumn5">
        <xp:viewColumnHeader value="Jonra"
            id="viewColumnHeader5">
        </xp:viewColumnHeader>
    </xp:viewColumn>
    <xp:viewColumn columnName="Seasonal" id="viewColumn6">
        <xp:viewColumnHeader value="Seasonal"
            id="viewColumnHeader6">
        </xp:viewColumnHeader>
    </xp:viewColumn>
    <xp:viewColumn id="viewColumn7" columnName="Location"
        style="width:75px">
        <xp:this.facets>
            <xp:viewColumnHeader xp:key="header"
                id="viewColumnHeader7" value="Location" style="width:750px">
            </xp:viewColumnHeader>
        </xp:this.facets>
    </xp:viewColumn>
    <xp:viewColumn id="viewColumn8" columnName="NumberOfCopies"
        style="width:10.0px">
        <xp:this.facets>
            <xp:viewColumnHeader xp:key="header"
                id="viewColumnHeader8" value="Number Of Copies"
                style="width:750px">
            </xp:viewColumnHeader>
        </xp:this.facets>
    </xp:viewColumn>
 </xp:viewPanel>

 <!-- DIALOG1-->

 <xe:dialog id="dialog1" title="Details">
    <xe:dialogContent id="dialogContent1">
        <xp:panel
            style="width:698.0px;background-
 image:url(Music1+Big+twice+Faded.png);background-repeat:no-
 repeat;background-position:center center;border-
 color:rgb(57,88,137);border-style:solid;border-
 width:thin;color:rgb(33,51,80)">

            <xp:this.data>
                <xp:dominoDocument var="document1"
                    formName="MusicForm" action="editDocument"
                    documentId="#{javascript:viewScope.UNID}"
                    ignoreRequestParams="true">
                </xp:dominoDocument>
            </xp:this.data>
            <xp:table>
                <xp:tr>
                    <xp:td>
                        <xp:label value="Music Type:"
                            id="musicType_Label1" for="musicType1"
                            style="font-weight:bold">
                        </xp:label>
                        <xp:br></xp:br>
                        <xp:inputText value="#{document1.MusicType}"
                            id="musicType1">
                        </xp:inputText>
                    </xp:td>
                    <xp:td>

                        <xp:label value="Title:" id="title_Label1"
                            for="title1" style="font-weight:bold">
                        </xp:label>
                        <xp:br></xp:br>
                        <xp:inputText value="#{document1.Title}"
                            id="title1">
                        </xp:inputText>
                    </xp:td>
                    <xp:td>
                        <xp:label value="Composed By:"
                            id="composedBy_Label1" for="composedBy1"
                            style="font-weight:bold">
                        </xp:label>
                        <xp:br></xp:br>
                        <xp:inputText value="#{document1.ComposedBy}"
                            id="composedBy1">
                        </xp:inputText>
                    </xp:td>
                    <xp:td>
                        <xp:label value="Arranged By:"
                            id="arrangedBy_Label1" for="arrangedBy1"
                            style="font-weight:bold">
                        </xp:label>
                        <xp:br></xp:br>
                        <xp:inputText value="#{document1.ArrangedBy}"
                            id="arrangedBy1">
                        </xp:inputText>
                    </xp:td>
                </xp:tr>
                <xp:tr>
                    <xp:td>

                        <xp:label value="Jonra:" id="jonra_Label1"
                            for="jonra1" style="font-weight:bold">
                        </xp:label>
                        <xp:br></xp:br>
                        <xp:inputText value="#{document1.Jonra}"
                            id="jonra1">
                        </xp:inputText>
                    </xp:td>
                    <xp:td>

                        <xp:label value="Season:"
                            id="seasonal_Label1" for="seasonal1"
 style="font-weight:bold">
                        </xp:label>
                        <xp:br></xp:br>
                        <xp:inputText value="#{document1.Seasonal}"
                            id="seasonal1">
                        </xp:inputText>
                    </xp:td>
                    <xp:td>
                        <xp:label value="Location:"
                            id="location_Label1" for="location1"
 style="font-weight:bold">
                        </xp:label>
                        <xp:br></xp:br>
                        <xp:inputText value="#{document1.Location}"
                            id="location1">
                        </xp:inputText>
                    </xp:td>
                    <xp:td>
                        <xp:label value="Number Of Copies:"
                            id="numberOfCopies_Label1" for="numberOfCopies1"
                            style="font-weight:bold">
                        </xp:label>
                        <xp:br></xp:br>
                        <xp:inputText
                            value="#{document1.NumberOfCopies}"
 id="numberOfCopies1">
                        </xp:inputText>
                    </xp:td>
                </xp:tr>
                <xp:tr>
                    <xp:td colspan="4">

                        <xp:messages layout="table"
                            style="color:red" id="messages1">
                        </xp:messages>
                    </xp:td>

                </xp:tr>
                <xp:tr>
                    <xp:td colspan="4">
                        <xp:button id="button5" value="Save">

                            <xp:this.rendered><![CDATA[#
 {javascript:if(document1.isNewNote()){
 false
 }else{
 true
 }}]]></xp:this.rendered>
                            <xp:eventHandler event="onclick"
                                submit="true" refreshMode="complete"
 id="eventHandler5">
                                <xp:this.action>

                                    <xp:actionGroup>
                                        <xp:executeScript
                                            script="#
 {javascript:document1.save()}">
                                        </xp:executeScript>
                                        <xp:openPage
                                            name="/Main.xsp">
                                        </xp:openPage>
                                    </xp:actionGroup>
                                </xp:this.action>
                            </xp:eventHandler>
                        </xp:button>
                        <xp:button id="button4" value="Submit">
                            <xp:this.rendered><![CDATA[#
 {javascript:if(document1.isNewNote()){
 true
 }else{
 false
}}]]></xp:this.rendered>
                            <xp:eventHandler event="onclick"
                                submit="true" refreshMode="complete"
immediate="false"
                                save="true" id="eventHandler2">
                                <xp:this.action>
                                    <xp:actionGroup>
                                        <xp:openPage
                                            name="/Main.xsp">
                                        </xp:openPage>
                                    </xp:actionGroup>
                                </xp:this.action>
                            </xp:eventHandler>
                        </xp:button>
                        <xp:button value="Edit" id="button2"
                            rendered="#
 {javascript:!document1.isEditable()}">
                            <xp:eventHandler event="onclick"
                                submit="true" refreshMode="complete"
id="eventHandler3">
                                <xp:this.action>
                                    <xp:changeDocumentMode
                                        mode="edit" var="document1">
                                    </xp:changeDocumentMode>
                                </xp:this.action>
                            </xp:eventHandler>
                        </xp:button>
                        <xp:button id="button3">
                            <xp:this.value><![CDATA[#
 {javascript:if(document1.isEditable()){
 "Cancel"
 }else{
 "Close"
 }}]]></xp:this.value>
                            <xp:eventHandler event="onclick"
                                submit="true" refreshMode="complete"
immediate="true"
                                save="false" id="eventHandler4">
                                <xp:this.action>
                                    <xp:openPage name="/Main.xsp">
                                    </xp:openPage>
                                </xp:this.action>
                            </xp:eventHandler>
                        </xp:button>
                    </xp:td>
                </xp:tr>
            </xp:table>
        </xp:panel>
    </xe:dialogContent>
</xe:dialog>
</xp:view>

What am I doing wrong?
MJ

A: "Maximum Internet name and password" on the ACL's Advanced tab can limit access. If that's set to e.g. Reader, the highest access permitted for the application on the web will be Reader access, regardless of the individual's ACL settings. Author would only allow them to create, not edit existing documents that do not have an Authors field etc.

redpajama_set_name: RedPajamaStackExchange


text:
Q: I need to combine multiple lines starting with the same ID I have multiple lines in a text file that I need to combine together. The file is about 200 million lines long, so opening it with Excel and using their built-in tools is out of the picture.
The first set of lines looks like this:
1,example@gmail.com,Username
3,example@gmail.com,Username
4,example@gmail.com,Username
5,example@gmail.com,Username
9,example@gmail.com,Username
10,example@gmail.com,Username

Second set which I want to add at the last line of the first set is:
1,$2a$10$gdsZkf62vUfwHQX8pUGe2.7zqvBvcIPWseaJmboJw3U2sxDj18y5q
3,$2a$10$gdsZkf62vUfwHQX8pUGe2.7zqvBvcIPWseaJmboJw3U2sxDj18y5q
4,$2a$10$gdsZkf62vUfwHQX8pUGe2.7zqvBvcIPWseaJmboJw3U2sxDj18y5q
5,$2a$10$gdsZkf62vUfwHQX8pUGe2.7zqvBvcIPWseaJmboJw3U2sxDj18y5q
9,$2a$10$gdsZkf62vUfwHQX8pUGe2.7zqvBvcIPWseaJmboJw3U2sxDj18y5q
10,$2a$10$gdsZkf62vUfwHQX8pUGe2.7zqvBvcIPWseaJmboJw3U2sxDj18y5q

If anyone has experience with this, I'd love some help

A: Code
Regex
^(\d+),(.*$)(?=[\s\S]*^\1,(.*))

Formatting output
$1,$2,$3

Results
Input
1,example@gmail.com,Username
3,example@gmail.com,Username
4,example@gmail.com,Username
5,example@gmail.com,Username
9,example@gmail.com,Username
10,example@gmail.com,Username
1,$2a$10$gdsZkf62vUfwHQX8pUGe2.7zqvBvcIPWseaJmboJw3U2sxDj18y5q
3,$2a$10$gdsZkf62vUfwHQX8pUGe2.7zqvBvcIPWseaJmboJw3U2sxDj18y5q
4,$2a$10$gdsZkf62vUfwHQX8pUGe2.7zqvBvcIPWseaJmboJw3U2sxDj18y5q
5,$2a$10$gdsZkf62vUfwHQX8pUGe2.7zqvBvcIPWseaJmboJw3U2sxDj18y5q
9,$2a$10$gdsZkf62vUfwHQX8pUGe2.7zqvBvcIPWseaJmboJw3U2sxDj18y5q
10,$2a$10$gdsZkf62vUfwHQX8pUGe2.7zqvBvcIPWseaJmboJw3U2sxDj18y5q

Output
1,example@gmail.com,Username,$2a$10$gdsZkf62vUfwHQX8pUGe2.7zqvBvcIPWseaJmboJw3U2sxDj18y5q
3,example@gmail.com,Username,$2a$10$gdsZkf62vUfwHQX8pUGe2.7zqvBvcIPWseaJmboJw3U2sxDj18y5q
4,example@gmail.com,Username,$2a$10$gdsZkf62vUfwHQX8pUGe2.7zqvBvcIPWseaJmboJw3U2sxDj18y5q
5,example@gmail.com,Username,$2a$10$gdsZkf62vUfwHQX8pUGe2.7zqvBvcIPWseaJmboJw3U2sxDj18y5q
9,example@gmail.com,Username,$2a$10$gdsZkf62vUfwHQX8pUGe2.7zqvBvcIPWseaJmboJw3U2sxDj18y5q
10,example@gmail.com,Username,$2a$10$gdsZkf62vUfwHQX8pUGe2.7zqvBvcIPWseaJmboJw3U2sxDj18y5q

Explanation

*

*^ Assert position at the start of a line

*(\d+) Capture one or more digits into capture group 1

*, Match the comma character , literally

*(.*$) Capture any number of any character (except newline characters) until the asserted position at the end of the line (asserting end of line position dramatically reduces steps) into capture group 2

*(?=[\s\S]*^\1,(.*)) Positive lookahead asserting what follows matches

*

*[\s\S]* Match any number of any character (\s: any whitespace character; \S: any non-whitespace character)

*^ Assert position at the start of a line

*\1 Matches the same text as most recently matched by the 1st capturing group

*, Matches the comma character , literally

*(.*) Capture any number of any character into capture group 3


text:
Q: FileNotFoundException on submitting Spark Jobs to remote I've created an environment where I've set up 3 Docker containers, 1 for Airflow using the puckel/docker-airflow image with spark and hadoop additionally installed. The other two containers are basically imitating spark master and worker (used gettyimages/spark Docker image to create this). All 3 containers are connected to each other via a bridge network, so all containers are able to communicate with each other.
What I'm trying to do next is to submit spark job from the Airflow container to the Spark cluster (master).
As an initial example, I'm using the wordcount sample script. I created a sample.txt file in the airflow container at path usr/local/airflow/sample.txt. I've bashed into the Airflow container and I'm using the command given below to run the wordcount.py on spark master located at the ip which I found after inspecting the bridge network.
spark-submit --master spark://ipaddress:7077 --files usr/local/airflow/sample.txt /opt/spark-2.4.1/examples/src/main/python/wordcount.py sample.txt
After submitting the script, from the logs, I can see that a connection has been established with the master (from airflow container), and it also copied the file specified by --files to the master and worker, but then it just errors out saying,
java.io.FileNotFoundException: File file:/usr/local/airflow/sample.txt does not exist
As per my understanding (could be wrong), but when we specify files to copy to master using --files you can access it directly via the file name (sample.txt in my case). So what I'm trying to figure out is if a job has been submitted and the file has been copied to master, then why is it searching in the location file:/usr/local/airflow/sample.txt? How do I make it refer to the correct path?
I apologize as this question has been asked a couple of times, but I've read all the related question on stackoverflow, but I'm still unable to resolve this. I'd really appreciate y'alls help on this.
Thanks.
The full log below,
user@machine:/usr/local/airflow# spark-submit --master spark://172.22.0.2:7077 --files sample.txt /opt/spark-2.4.1/examples/src/main/python/wordcount.py ./sample.txt
20/07/25 03:23:34 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
20/07/25 03:23:35 INFO SparkContext: Running Spark version 2.4.1
20/07/25 03:23:35 INFO SparkContext: Submitted application: PythonWordCount
20/07/25 03:23:35 INFO SecurityManager: Changing view acls to: root
20/07/25 03:23:35 INFO SecurityManager: Changing modify acls to: root
20/07/25 03:23:35 INFO SecurityManager: Changing view acls groups to:
20/07/25 03:23:35 INFO SecurityManager: Changing modify acls groups to:
20/07/25 03:23:35 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
20/07/25 03:23:35 INFO Utils: Successfully started service 'sparkDriver' on port 33457.
20/07/25 03:23:35 INFO SparkEnv: Registering MapOutputTracker
20/07/25 03:23:36 INFO SparkEnv: Registering BlockManagerMaster
20/07/25 03:23:36 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
20/07/25 03:23:36 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
20/07/25 03:23:36 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-dd1957de-6907-484d-a3d8-2b3b88e0c7ca
20/07/25 03:23:36 INFO MemoryStore: MemoryStore started with capacity 366.3 MB
20/07/25 03:23:36 INFO SparkEnv: Registering OutputCommitCoordinator
20/07/25 03:23:36 INFO Utils: Successfully started service 'SparkUI' on port 4040.
20/07/25 03:23:36 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://0508a77fcaad:4040
20/07/25 03:23:37 INFO SparkContext: Added file file:///usr/local/airflow/sample.txt at spark://0508a77fcaad:33457/files/sample.txt with timestamp 1595647417081
20/07/25 03:23:37 INFO Utils: Copying /usr/local/airflow/sample.txt to /tmp/spark-f9dfe6ee-22d7-4747-beab-9450fc1afce0/userFiles-74f8cfe4-8a19-4d2e-8fa1-1f0bd1f0ef12/sample.txt
20/07/25 03:23:37 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://172.22.0.2:7077...
20/07/25 03:23:37 INFO TransportClientFactory: Successfully created connection to /172.22.0.2:7077 after 32 ms (0 ms spent in bootstraps)
20/07/25 03:23:38 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20200725032338-0003
20/07/25 03:23:38 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 45057.
20/07/25 03:23:38 INFO NettyBlockTransferService: Server created on 0508a77fcaad:45057
20/07/25 03:23:38 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
20/07/25 03:23:38 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20200725032338-0003/0 on worker-20200725025003-172.22.0.4-8881 (172.22.0.4:8881) with 2 core(s)
20/07/25 03:23:38 INFO StandaloneSchedulerBackend: Granted executor ID app-20200725032338-0003/0 on hostPort 172.22.0.4:8881 with 2 core(s), 1024.0 MB RAM
20/07/25 03:23:38 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 0508a77fcaad, 45057, None)
20/07/25 03:23:38 INFO BlockManagerMasterEndpoint: Registering block manager 0508a77fcaad:45057 with 366.3 MB RAM, BlockManagerId(driver, 0508a77fcaad, 45057, None)
20/07/25 03:23:38 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 0508a77fcaad, 45057, None)
20/07/25 03:23:38 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 0508a77fcaad, 45057, None)
20/07/25 03:23:38 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20200725032338-0003/0 is now RUNNING
20/07/25 03:23:38 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.020/07/25 03:23:38 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/usr/local/airflow/spark-warehouse').
20/07/25 03:23:38 INFO SharedState: Warehouse path is 'file:/usr/local/airflow/spark-warehouse'.
20/07/25 03:23:40 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
20/07/25 03:23:47 INFO FileSourceStrategy: Pruning directories with:
20/07/25 03:23:47 INFO FileSourceStrategy: Post-Scan Filters:
20/07/25 03:23:47 INFO FileSourceStrategy: Output Data Schema: struct<value: string>
20/07/25 03:23:47 INFO FileSourceScanExec: Pushed Filters:
20/07/25 03:23:51 INFO CodeGenerator: Code generated in 2187.926234 ms
20/07/25 03:23:53 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 220.9 KB, free 366.1 MB)
20/07/25 03:23:55 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 20.8 KB, free 366.1 MB)
20/07/25 03:23:55 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 0508a77fcaad:45057 (size: 20.8 KB, free: 366.3 MB)
20/07/25 03:23:55 INFO SparkContext: Created broadcast 0 from javaToPython at NativeMethodAccessorImpl.java:0
20/07/25 03:23:55 INFO FileSourceScanExec: Planning scan with bin packing, max size: 4194304 bytes, open cost is considered as scanning 4194304 bytes.
20/07/25 03:23:57 INFO SparkContext: Starting job: collect at /opt/spark-2.4.1/examples/src/main/python/wordcount.py:40
20/07/25 03:23:58 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.22.0.4:59324) with ID 0
20/07/25 03:23:58 INFO DAGScheduler: Registering RDD 5 (reduceByKey at /opt/spark-2.4.1/examples/src/main/python/wordcount.py:39)
20/07/25 03:23:58 INFO DAGScheduler: Got job 0 (collect at /opt/spark-2.4.1/examples/src/main/python/wordcount.py:40) with 1 output partitions
20/07/25 03:23:58 INFO DAGScheduler: Final stage: ResultStage 1 (collect at /opt/spark-2.4.1/examples/src/main/python/wordcount.py:40)
20/07/25 03:23:58 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)
20/07/25 03:23:58 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)
20/07/25 03:23:58 INFO DAGScheduler: Submitting ShuffleMapStage 0 (PairwiseRDD[5] at reduceByKey at /opt/spark-2.4.1/examples/src/main/python/wordcount.py:39), which has no missing parents
20/07/25 03:23:58 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 15.2 KB, free 366.0 MB)
20/07/25 03:23:58 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 9.1 KB, free 366.0 MB)
20/07/25 03:23:58 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 0508a77fcaad:45057 (size: 9.1 KB, free: 366.3 MB)
20/07/25 03:23:58 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1161
20/07/25 03:23:58 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 0 (PairwiseRDD[5] at reduceByKey at /opt/spark-2.4.1/examples/src/main/python/wordcount.py:39) (first 15 tasks are for partitions Vector(0))
20/07/25 03:23:58 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
20/07/25 03:23:58 INFO BlockManagerMasterEndpoint: Registering block manager 172.22.0.4:45435 with 366.3 MB RAM, BlockManagerId(0, 172.22.0.4, 45435, None)
20/07/25 03:23:58 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 172.22.0.4, executor 0, partition 0, PROCESS_LOCAL, 8307 bytes)
20/07/25 03:24:03 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 172.22.0.4:45435 (size: 9.1 KB, free: 366.3 MB)
20/07/25 03:24:09 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.22.0.4:45435 (size: 20.8 KB, free: 366.3 MB)
20/07/25 03:24:11 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, 172.22.0.4, executor 0): java.io.FileNotFoundException: File file:/usr/local/airflow/sample.txt does not exist
It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved.
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:127)
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:177)
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
        at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)
        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
        at org.apache.spark.api.python.SerDeUtil$AutoBatchedPickler.hasNext(SerDeUtil.scala:153)
        at scala.collection.Iterator$class.foreach(Iterator.scala:891)
        at org.apache.spark.api.python.SerDeUtil$AutoBatchedPickler.foreach(SerDeUtil.scala:148)
        at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
        at org.apache.spark.api.python.PythonRunner$$anon$2.writeIteratorToStream(PythonRunner.scala:557)
        at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run$1.apply(PythonRunner.scala:345)
        at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
        at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:194)

20/07/25 03:24:11 INFO TaskSetManager: Starting task 0.1 in stage 0.0 (TID 1, 172.22.0.4, executor 0, partition 0, PROCESS_LOCAL, 8307 bytes)
20/07/25 03:24:11 INFO TaskSetManager: Lost task 0.1 in stage 0.0 (TID 1) on 172.22.0.4, executor 0: java.io.FileNotFoundException (File file:/usr/local/airflow/sample.txt does not exist
It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved.) [duplicate 1]
20/07/25 03:24:11 INFO TaskSetManager: Starting task 0.2 in stage 0.0 (TID 2, 172.22.0.4, executor 0, partition 0, PROCESS_LOCAL, 8307 bytes)
20/07/25 03:24:12 INFO TaskSetManager: Lost task 0.2 in stage 0.0 (TID 2) on 172.22.0.4, executor 0: java.io.FileNotFoundException (File file:/usr/local/airflow/sample.txt does not exist
It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved.) [duplicate 2]
20/07/25 03:24:12 INFO TaskSetManager: Starting task 0.3 in stage 0.0 (TID 3, 172.22.0.4, executor 0, partition 0, PROCESS_LOCAL, 8307 bytes)
20/07/25 03:24:12 INFO TaskSetManager: Lost task 0.3 in stage 0.0 (TID 3) on 172.22.0.4, executor 0: java.io.FileNotFoundException (File file:/usr/local/airflow/sample.txt does not exist
It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved.) [duplicate 3]
20/07/25 03:24:12 ERROR TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting job
20/07/25 03:24:12 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
20/07/25 03:24:12 INFO TaskSchedulerImpl: Cancelling stage 0
20/07/25 03:24:12 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage cancelled
20/07/25 03:24:12 INFO DAGScheduler: ShuffleMapStage 0 (reduceByKey at /opt/spark-2.4.1/examples/src/main/python/wordcount.py:39) failed in 13.690 s due to Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, 172.22.0.4, executor 0): java.io.FileNotFoundException: File file:/usr/local/airflow/sample.txt does not exist
It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved.
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:127)
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:177)
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
        at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)
        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
        at org.apache.spark.api.python.SerDeUtil$AutoBatchedPickler.hasNext(SerDeUtil.scala:153)
        at scala.collection.Iterator$class.foreach(Iterator.scala:891)
        at org.apache.spark.api.python.SerDeUtil$AutoBatchedPickler.foreach(SerDeUtil.scala:148)
        at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
        at org.apache.spark.api.python.PythonRunner$$anon$2.writeIteratorToStream(PythonRunner.scala:557)
        at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run$1.apply(PythonRunner.scala:345)
        at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
        at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:194)

Driver stacktrace:
20/07/25 03:24:12 INFO DAGScheduler: Job 0 failed: collect at /opt/spark-2.4.1/examples/src/main/python/wordcount.py:40, took 14.579961 s
Traceback (most recent call last):
  File "/opt/spark-2.4.1/examples/src/main/python/wordcount.py", line 40, in <module>
    output = counts.collect()
  File "/opt/spark-2.4.1/python/lib/pyspark.zip/pyspark/rdd.py", line 816, in collect
  File "/opt/spark-2.4.1/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
  File "/opt/spark-2.4.1/python/lib/pyspark.zip/pyspark/sql/utils.py", line 63, in deco
  File "/opt/spark-2.4.1/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, 172.22.0.4, executor 0): java.io.FileNotFoundException: File file:/usr/local/airflow/sample.txt does not exist
It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved.
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:127)
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:177)
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
        at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)
        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
        at org.apache.spark.api.python.SerDeUtil$AutoBatchedPickler.hasNext(SerDeUtil.scala:153)
        at scala.collection.Iterator$class.foreach(Iterator.scala:891)
        at org.apache.spark.api.python.SerDeUtil$AutoBatchedPickler.foreach(SerDeUtil.scala:148)
        at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
        at org.apache.spark.api.python.PythonRunner$$anon$2.writeIteratorToStream(PythonRunner.scala:557)
        at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run$1.apply(PythonRunner.scala:345)
        at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
        at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:194)

Driver stacktrace:
        at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1889)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1877)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1876)
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
        at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1876)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
        at scala.Option.foreach(Option.scala:257)
        at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:926)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2110)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2059)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2048)
        at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
        at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:2126)
        at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:945)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
        at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
        at org.apache.spark.rdd.RDD.collect(RDD.scala:944)
        at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:166)
        at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.GatewayConnection.run(GatewayConnection.java:238)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.FileNotFoundException: File file:/usr/local/airflow/sample.txt does not exist
It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved.
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:127)
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:177)
        at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
        at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)
        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
        at org.apache.spark.api.python.SerDeUtil$AutoBatchedPickler.hasNext(SerDeUtil.scala:153)
        at scala.collection.Iterator$class.foreach(Iterator.scala:891)
        at org.apache.spark.api.python.SerDeUtil$AutoBatchedPickler.foreach(SerDeUtil.scala:148)
        at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
        at org.apache.spark.api.python.PythonRunner$$anon$2.writeIteratorToStream(PythonRunner.scala:557)
        at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run$1.apply(PythonRunner.scala:345)
        at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
        at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:194)

20/07/25 03:24:13 INFO SparkContext: Invoking stop() from shutdown hook
20/07/25 03:24:13 INFO SparkUI: Stopped Spark web UI at http://0508a77fcaad:4040
20/07/25 03:24:13 INFO StandaloneSchedulerBackend: Shutting down all executors
20/07/25 03:24:13 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
20/07/25 03:24:16 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
20/07/25 03:24:16 INFO MemoryStore: MemoryStore cleared
20/07/25 03:24:16 INFO BlockManager: BlockManager stopped
20/07/25 03:24:16 INFO BlockManagerMaster: BlockManagerMaster stopped
20/07/25 03:24:16 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
20/07/25 03:24:16 INFO SparkContext: Successfully stopped SparkContext
20/07/25 03:24:16 INFO ShutdownHookManager: Shutdown hook called
20/07/25 03:24:16 INFO ShutdownHookManager: Deleting directory /tmp/spark-2dfb2222-d56c-4ee1-ab62-86e71e5e751b
20/07/25 03:24:16 INFO ShutdownHookManager: Deleting directory /tmp/spark-f9dfe6ee-22d7-4747-beab-9450fc1afce0
20/07/25 03:24:16 INFO ShutdownHookManager: Deleting directory /tmp/spark-f9dfe6ee-22d7-4747-beab-9450fc1afce0/pyspark-2ee74d07-6606-4edc-8420-fe46212c50e5

A: Change your spark-submit like below for submitting your spark job.
spark-submit \
--master spark://ipaddress:7077 \
--deploy-mode cluster # add this if you want to pass file name to wordcount.py
--files usr/local/airflow/sample.txt \
/opt/spark-2.4.1/examples/src/main/python/wordcount.py sample.txt

OR
spark-submit \
--master spark://ipaddress:7077 \
/opt/spark-2.4.1/examples/src/main/python/wordcount.py /usr/local/airflow/sample.txt


text:
Q: S3 to Redshift - Cross Account encrypted data issue I did the following tasks between two of my accounts ( Dev and Prod ) and I am not able to copy data from S3 to Redshift in my Prod Account.
Task 1:

*

*I have some data in my Dev Account under S3

*This data is encrypted using KMS

*I wanted to move the same data to Prod Account

*I started a EC2 Server and executed sync command to move data from Dev to Prod ( My Bad I did not focused on decryption while syncing )

*My data is available in Prod Account ( around 500 GB )

Task 2:
Now I wanted to move this S3 data ( Which is copied from Dev ) to
Redshift , I executed the following command using Redshift Query
Editor.
Command:
COPY table1(wre,sdf,sfd,sd,fs,df,sdfs,dfsdf,sd,fs,dfsdf,sdf,qwer,wer,wer)
FROM 's3://mydata/csv'
IAM_ROLE 'arn:aws:iam::myrole'
DELIMITER '|' IGNOREHEADER 1 TIMEFORMAT 'auto' REMOVEQUOTES NULL AS 'NULL'
EMPTYASNULL TRIMBLANKS ROUNDEC;

But I encountered the following error :
[Amazon](500310) Invalid operation: S3ServiceException:Access Denied,Status 403,
Error AccessDenied,Rid 70FE39F0982E9706,ExtRid Pxs9xdm+Plb2fvgySsuVq0UWLV2UCwMJ+qrGe0jazom346t4szXILbHma5SX1ZrIq4kPJA8q5Ws=,
CanRetry 1 Details: -----------------------------------------------
error: S3ServiceException:Access Denied,Status 403,Error AccessDenied,
Rid 70FE39F0982E9706,ExtRid Pxs9xdm+Plb2fvgySsuVq0UWLV2UCwMJ+qrGe0jazom346t4szXILbHma5SX1ZrIq4kPJA8q5Ws=,
CanRetry 1 code: 8001 context: S3 key being read :

I thought that since the data is encrypted I am facing this issue. So in order to overcome this I tried to allow cross account access to the KMS.
Task 3:
I gone through this document and followed the procedure. Here is the steps for the same and the policy which I have created .

*

*I created a IAM Policy ( described below ) in Production Account

*Attached this policy to my Role (Which is used in copy statement above) for Redshift copy in Production Account

Policy :
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowUseOfCMKInAccount444455556666",
      "Effect": "Allow",
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:DescribeKey"
      ],
      "Resource": "arn:aws:kms:us-east-2:XXXXXXXXXX:key/9821114k-mfdfki-7e8e"
    },
    {
      "Sid": "AllowUseofCMKToCreateEncryptedResourcesInAccount444455556666",
      "Effect": "Allow",
      "Action": "kms:CreateGrant",
      "Resource": "arn:aws:kms:us-east-2:XXXXXXXXXX:key/9821114k-mfdfki-7e8e",
      "Condition": {
        "Bool": {
          "kms:GrantIsForAWSResource": true
        }
      }
    }
  ]
}

Where XXXXXXXXXX is dev account number and arn is Dev Account Key arn.
What other changes I will have to make to get this running and get my data into Redshift.
It will be great if someone could help me understand the background behaviour as well.
NOTE : I am using Customer Managed Key in Dev Account in KMS.

A: This doesn't seem like a KMS error.
You need to give Redshift permission to access the data in S3 Bucket.

The user account identified by the credentials must have LIST and GET
  access to the Amazon S3 bucket. If the user does not have sufficient
  privileges, you will receive the following error message:
ERROR: S3ServiceException:Access Denied,Status 403,Error AccessDenied

For more info see this link: https://docs.aws.amazon.com/redshift/latest/dg/s3serviceexception-error.html


text:
Q: Privoxy, Tor and cURL using Zend HTTP I am really stuck in this, and need expert help. Let me explain what I am trying achieve and the setup. I  had a script which posts over a https form using Zend_Http_Client. On the server setup I have tor & privoxy running. Everything just worked fine, but now I need to make the code more scalable by running  multiple instances of tor & privoxy on the same server.
Hence I shifted the Adapter for Zend from Zend_Http_Client_Adapter_Curl to Zend_Http_Client_Adapter_Proxy . But after changing the adapter I have bumped into a strange error which says - 400 Invalid header received from client and when I dump the object of Zend client, I see the following -
MyWebClientResponse::__set_state(array(
   'json' => NULL,
   'version' => '1.1',
   'code' => 400,
   'message' => 'Invalid header received from client',
   'headers' =>
  array (
    'Proxy-agent' => 'Privoxy 3.0.19',
    'Content-type' => 'text/plain',
    'Connection' => 'close',
  ),
   'body' => 'Invalid header received from client.
',
))

I do not understand what is that I am doing wrong. The code is done in Yii Framework so it is hard to share all the classes and Models, but I am sharing the main parts of the code, which are responsible for this -
$client = MyWebClient::factory();
$adapter = $client->getAdapter();
$adapter->setConfig(array('timeout' => 120,
'proxy_host' => 'localhost',
'proxy_port' => 8124
));
$client->setAdapter($adapter);
$client->setCookieJar(true);
$client->setParameterPost(array(
'name' => 'firstname',
'password' => 'password,
'login' => 'home'
));
$response = $client->setUri('https://www.domain.com/post.php')->requestApi('POST', false);

Here's the constructor of the class MyWebClient, just in case it it required, all other methods are standard.
  static public function factory($new = false)
    {
        if (!isset(self::$client))
        {
            self::$client = new MyWebClient();
            self::$client->setConfig(array(
                'adapter' => 'Zend_Http_Client_Adapter_Proxy',
//              'proxy_host' => 'localhost',
//              'proxy_port' => 8118,
                'persistent' => false,
                'timeout' => 120
            ));
        }
        return self::$client;
    }

The headers are being set in the requestAPI method and the snippet is -
 $headers = array(
            'X-PHX' => 'true',
            'X-Requested-With' => 'XMLHttpRequest',
            'Referer' => 'https://domain.com/index.html',
            'User-Agent' => self::getRandomUserAgent()
        );
        $this->setHeaders($headers);

I would really appreciate help in this regard. Think it's privoxy which is not letting go the request out of the server.
Sachin

A: Looks like you may need to wrap the user-agent with quotes. Try this when setting the headers and see if that fixes the problem.
$headers = array(
        'X-PHX' => 'true',
        'X-Requested-With' => 'XMLHttpRequest',
        'Referer' => 'https://domain.com/index.html',
        'User-Agent' => "'".self::getRandomUserAgent()."'"
    );
    $this->setHeaders($headers);

A: Finally I decided to use the native cURL library and the cookieJar which comes with it. It worked like expected.
Cheers,
Sachin


text:
Q: Using Pest.rs, how can I avoid 'peek was called on empty stack' if the PUSH is optional? Pest.rs has the ability to push and peek onto a stack. This is useful when a delimiter is given by the user like custom-quoting found in Perl, and PostgreSQL (double dollar syntax). How can I do this if the item may not be on the stack. For example, the Exim config file states,

It is also possible to use newline and other control characters (those with code values less than 32, plus DEL) as separators in lists. Such separators must be provided literally at the time the list is processed. For options that are string-expanded, you can write the separator using a normal escape sequence. This will be processed by the expander before the string is interpreted as a list. For example, if a newline-separated list of domains is generated by a lookup, you can process it directly by a line such as this:
domains = <\n ${lookup mysql{.....}}

You can see here the token <\n which overrides the default separator of : is optional. You can see here that the syntax is essentially,
list_sep        = ${ "<" ~ (!WHITE_SPACE ~ ANY) }
list            = {
  list_type
  ~ key
  ~ "="
  ~ PUSH(list_sep)?
  ~ !(PEEK ~ ANY)+
  ~ (PEEK ~ !(PEEK ~ ANY))*
}

But when I run that I get,

peek was called on empty stack

Is there anyway to provide for this operation with PEG such that the list is presented to the user as a string, but as an actual list of element tokens?

A: One method I've started doing is making sure the PUSH always happens. It even seems like an opitional PUSH should be a compiler error,
Rather than
~ PUSH(list_sep)?

Do this,
~ PUSH(list_sep?)


text:
Q: Gdx.files.internal not using android assets path Whenever I try to use Gdx.files.internal to get a FileHandle, I am getting a FileNotFoundException like follows:
    <PATH TO MY GAME>\desktop\xml\screen.xml (The system cannot find the path specified)

I set my project up using the libgdx project setup tool, and it seems that my asset folder within my desktop project is the same as my android project (they're linked). However, when I go to my desktop project there is a bin folder with my assets. If I change the path to "/bin/..." then I can run the project without any exceptions. However, I feel this may be a problem if I export as a JAR.
Code:
    final FileHandle screenXML = Gdx.files.internal("xml/screen.xml");

A: My problem stemmed from the following (credit goes to the awesome and patient IRC community members for Libgdx):
    // final Document screenDoc = docBuilder.parse(screenXML.file());
    // Note: MUST use .read() here, not .file(), as we link from the android assets!
    final Document screenDoc = docBuilder.parse(screenXML.read());


text:
Q: How to get the benefit of INET framework in OMNET++? How could I get the benefit of Inet framework on omnet++ to build a wireless communication system without editing the existed source codes? (use the existed source code as a black box)
for example, I want to receive a msg at node B from node A, and then reply to node A in a different time with a new msg and different details of the scenario using for example TCPGenericServiceApp.

A: INET is a quite modular framework.
The OMNeT++ network is created using a omnetpp.ini file which is usually located in the according example folder.
If you now want to change something without changing the files included in INET you can try to create your own network.
Hence, you have to create your own omnetpp.ini in a new folder and than describe your network in that file.
Using this method, you can create your own network setup with network modules, already provided by INET.
If you also want to change the behavior of a certain simple module provided by INET you can create a new OMNeT++ project which is based on INET.
Then you can use all modules provided by INET as base classes.
To change their behavior, you have to create new subclasses from the INET base classes and override the according methods. Also you have to create new NED files for your new C++ classes.
Then you can use your newly defined network modules in your omnetpp.ini without the need of changing anything in the current INET repository.


text:
Q: Using unicode character in character literal As we know something like '═' will give a multi-char constant warning. The alternative is either to use 0x2550 or "\u2550". However, the latter requires the printing function to support the \u escape sequence. However I don't want to use hexadecimal everywhere. Is there a language construct that I can use that will allow me to write '=' when I really mean uint32_t c = 0x2550, i.e., '='u.

A: The \u sequence is interpreted by the C++ compiler, not by a "printing function."  However, you need to use a wide string (std::wstring) in order to store the character.  For example:
#include <iostream>
#include <string>

int main() {
    std::wstring s(L"\u2550");

    std::wcout << s.size() << std::endl
               << std::hex << static_cast<int>(s[0]) << std::dec << std::endl
               << s << std::endl;

    return 0;
}

The output shows that there is only one character in this string.

A:
" As we know something like '═' will give a multi-char constant warning.

No.

" The alternative is either to use 0x2550 or "\u2550".

No.

" However, the latter requires the printing function to support the \u escape sequence.

No.

" However I don't want to use hexadecimal everywhere. Is there a language construct that I can use that will allow me to write '═' when I really mean uint32_t c = 0x2550, i.e., '═'u.

yes.
uint32_t c = L'═';

with a source encoding that supports that character.
For characters not in the Basic Multilingual Plane of Unicode it gets more involved though, because a wchar_t in Windows is only 16 bits, as in original Unicode.


text:
Q: insert select issue mysqli php I got a very simple mysqli statement in php 7.1 on apache web-server:
INSERT INTO user SELECT * FROM user_geloescht WHERE id = '294930';

This should copy a user into a separate table for deleted users. But when I run the script, nothing happens to the database. When I copy the statement to my mariadb cli it works without any error.
I already looked for a similar "SHOW columns from user" and ... user_geloescht output but that doesnt fix it.
I would give you some debugging but the query result is just true -> so no error.
What can be wrong or what output can I supply to you?
Edit: as i wrote in the comments - it works when I perform an code-exit immediatly after the insert-select so it may be something about commiting.
A delete on the user-id in the user-table which is performed after the insert-select is also without any effect
I will keep you up-to date after I tried forced commits.
Putting PHP-Code isn't that easy caused by my class-structure and handling of arguments between some functions

A: this is the syntax:
INSERT INTO user (name)SELECT name FROM user WHERE id = 294930

assuming you only have a column in a user table called name. and I don't know what is that user_geloescht mean for. is that a table or a column name.

A: Oh I think i need more coffee :D.
The problem was that everythink worked as it should do but I handled it all wrong.
The insert-select is fine but my delete pointed on the wrong table, so i copied the user from user_geloescht-table to user-table but deleted the copied user on the user-table and not on user_geloescht-table.
Thanks to all your effort -> I will request some holiday :D


text:
Q: Number of rows on the basis of Foreign Key I have two tables class_tbl (id,class_name) and  student_tbl(id,class_id,student_name).
I want to display all class in a table format and want to show number of students  in each class.
How to do this in one mysql query?

A: Use this:
SELECT class_name, count(student_tbl.id) as number_of_students FROM class_tbl c INNER JOIN student_tbl s ON c.id=s.class_id GROUP BY class_id;

A: You have provided very little information in your question. Please take some time to read some of the question on stackoverflow which are closed / answered / up-voted and spend some time thinking about why they have merited such action.
Assuming that your "student_tbl" actually decomposes the N:M relationship between students and classes (otherwise, as Honeyboy Wilson says, it is impossible to resolve the query) all you need is a count of records in the table grouped by class_id. If you want to add the name of the class then its just a join. This is remedial level SQL and already covered in other answers.


text:
Q: Choosing a type for search results in C# I have a result set that will never exceed 500; the results that come back from the web-service are assigned to a search results object.
The data from the webservice is about 2mb; the bit I want to use is about a third of each record, so this allows me to cache and quickly manipulate it.
I want to be able to sort and filter the results with the least amount of overhead and as fast as possible so I used the VCSKICKS timing class to measure their performance
              Average        Total (10,000)
Type       Create  Sort      Create  Sort
HashSet    0.1579  0.0003    1579    3
IList      0.0633  0.0002    633     2
IQueryable 0.0072  0.0432    72      432

Measured in Seconds using http://www.vcskicks.com/algorithm-performance.php
I created the hashset through a for loop over the web-service response (adding to the hashset).
The List & IQueryable were created using LINQ.
Question
I can understand why HashSet takes longer to create (the foreach loop vs linq); but why would IQueryable take longer to sort than the other two; and finally is there a better way to assign the HashSet.
Thanks
Actual Program
public class Program
{
    private static AuthenticationHeader _authHeader;
    private static OPSoapClient _opSession;
    private static AccommodationSearchResponse _searchResults;

    private static HashSet<SearchResults> _myHash;
    private static IList<SearchResults> _myList;
    private static IQueryable<SearchResults> _myIQuery;

    static void Main(string[] args)
    {
        #region Setup WebService
        _authHeader = new AuthenticationHeader { UserName = "xx", Password = "xx" };
        _opSession = new OPSoapClient();

        #region Setup Search Results

        _searchResults = _opgSession.SearchCR(_authHeader, "ENG", "GBP", "GBR");

        #endregion Setup Search Results

        #endregion Setup WebService

        // HASHSET
        SpeedTester hashTest = new SpeedTester(TestHashSet);
        hashTest.RunTest();
        Console.WriteLine("- Hash Test \nAverage Running Time: {0}; Total Time: {1}",
            hashTest.AverageRunningTime,
            hashTest.TotalRunningTime);

        SpeedTester hashSortTest = new SpeedTester(TestSortingHashSet);
        hashSortTest.RunTest();
        Console.WriteLine("- Hash Sort Test \nAverage Running Time: {0}; Total Time: {1}",
            hashSortTest.AverageRunningTime,
            hashSortTest.TotalRunningTime);

        // ILIST
        SpeedTester listTest = new SpeedTester(TestList);
        listTest.RunTest();
        Console.WriteLine("- List Test \nAverage Running Time: {0}; Total Time: {1}",
            listTest.AverageRunningTime,
            listTest.TotalRunningTime);

        SpeedTester listSortTest = new SpeedTester(TestSortingList);
        listSortTest.RunTest();
        Console.WriteLine("- List Sort Test \nAverage Running Time: {0}; Total Time: {1}",
            listSortTest.AverageRunningTime,
            listSortTest.TotalRunningTime);

        // IQUERIABLE
        SpeedTester iqueryTest = new SpeedTester(TestIQueriable);
        iqueryTest.RunTest();
        Console.WriteLine("- iquery Test \nAverage Running Time: {0}; Total Time: {1}",
            iqueryTest.AverageRunningTime,
            iqueryTest.TotalRunningTime);

        SpeedTester iquerySortTest = new SpeedTester(TestSortableIQueriable);
        iquerySortTest.RunTest();
        Console.WriteLine("- iquery Sort Test \nAverage Running Time: {0}; Total Time: {1}",
            iquerySortTest.AverageRunningTime,
            iquerySortTest.TotalRunningTime);
    }

    static void TestHashSet()
    {
        var test = _searchResults.Items;

        _myHash = new HashSet<SearchResults>();

        foreach(var x in test)
        {
            _myHash.Add(new SearchResults
                                     {
                                         Ref = x.Ref,
                                         Price = x.StandardPrice
                                     });
        }

    }

    static void TestSortingHashSet()
    {
        var sorted = _myHash.OrderBy(s => s.Price);
    }

    static void TestList()
    {
        var test = _searchResults.Items;

        _myList = (from x in test
                   select new SearchResults
                              {
                                  Ref = x.Ref,
                                  Price = x.StandardPrice
                              }).ToList();
    }

    static void TestSortingList()
    {
        var sorted = _myList.OrderBy(s => s.Price);
    }

    static void TestIQueriable()
    {
        var test = _searchResults.Items;

        _myIQuery = (from x in test
                     select new SearchResults
                                {
                                    Ref = x.Ref,
                                    Price = x.StandardPrice
                                }).AsQueryable();

    }

    static void TestSortableIQueriable()
    {
        var sorted = _myIQuery.OrderBy(s => s.Price);
    }

}

=== Direct Output from console ===
- Hash Test
Average Running Time: 0.154; Total Time: 1540
- Hash Sort Test
Average Running Time: 0.0003; Total Time: 3
- List Test
Average Running Time: 0.0601; Total Time: 601
- List Sort Test
Average Running Time: 0.0002; Total Time: 2
- iquery Test
Average Running Time: 0.0003; Total Time: 3
- iquery Sort Test
Average Running Time: 0.0399; Total Time: 399
- IEnum Test
Average Running Time: 0.0002; Total Time: 2
- IEnum Sort Test
Average Running Time: 0.0001; Total Time: 1
Press any key to continue . . .

Obviously times vary based on what my machine was doing at the time, you can usually make some judgement based on the size of the difference but as the 'answer' states its unfair to judge IQueryable the same way.

A: I think IQueryable has deferred the creation until the sort was executed where the IList did the creation up front and then performed the sort on it.
Just like the IList, the HashSet creation was not deferred and therefore the the big up front cost.
So up front the HashSet and the IList have to be created which takes time which you record.  Then you record the sorting on them which is probably negligible.  The IQueryable deferred that creation time UNTIL you did the sort so the create and sort are all happening in the time for the sort.  The sort didn't really take longer for the IQueryable it just looks like it because the creation is happening at the same time in your testing.  I would suspect that all 3 sort's time differences are negligible.
I think either way it is fine to use any of them depending on how you want the output to be used or extended in the case of IQueryable.


text:
Q: prolog List Reverse %reverse_List
reverseList(H|T,ReversedList):-
    reverseListHelper(T,[H],ReversedList).
reverseListHelper([],Accumulator.Accumulator).
reverseListHelper([H|T],Accumulator,ReversedList):-
    reverseListHelper(T,[H|Accumulator],ReversedList).

I am beginner to prolog, what wrong with this code ?
it's just giving the output false
kindly explain thanks
I understand the theory of how recursive works in list reversing but not the code much,  if any one could explain line by line would be great thanks

A: One can look at a list in Prolog as essentially a FIFO (first-in/first-out) stack: you can examine, add or remove things only from the top/head: the remainder of the list is opaque.
Reversal of such a list, then, consists of repeatedly (and recursively) popping items from the source stack and pushing each such popped item onto a result stack.
And a common idiom in prolog is the use of a helper predicate with the same name, but different arity, where the additional argument(s) carry whatever extra state is required to solve the problem. In this case, the extra state needs is a list onto which we can push/prepend things as we go, thus building the result in reverse order.
That leads to an implementation of reverse_list/2 that looks like this:
reverse_list( Xs, Ys ) :- reverse_list( Xs , [] , Ys ) .

The helper that does all the work isn't much more complicated. There are just two cases:

*

*The limiting case, where the source list is the empty list, and

*The general case, where the source list is non-empty

The limiting case is easy: if the source list is exhausted, the accumulator, whatever it might contain, is the reversed list. That gives us
reverse_list( [] , Zs, Zs ) .

The general case merely involves

*

*removing the head of the source list,

*prepending it to the accumulator, and

*recursing down on the tails.

Which leads to this (you might note that we leave the final result alone until we hit the limiting case here)
reverse( [X|Xs] , [X|Ys] , Zs ) :- reverse(Xs,Ys,Zs)

Putting it all together, we get
% reverse/2 ----------------------------------------------------------
%
% reverse( Source, Reverse )
%
% Simply invoke the helper predicate, seeding the accumulator with the
% empty list.
%
% --------------------------------------------------------------------
reverse( Xs , Ys ) :- reverse(Xs,[],Ys) .

% reverse/3 ----------------------------------------------------------
%
% reverse( Source, Accumulator, Reversed )
%
% --------------------------------------------------------------------
reverse( []     ,    Zs  , Zs ) .  % source list empty: unify accumulator and result
reverse( [X|Xs] , [X|Ys] , Zs ) :- % non-empty source? Put X on the accumulator, and
  reverse(Xs,Ys,Zs) .              % - recurse down on the tails.


text:
Q: Code not evaluated in default argument value specification (At least some kind of) Ruby code is accepted and evaluated within the default value specification of a method. In below, "foo" * 3 is evaluated:
def bar baz = "foo" * 3; baz end
bar # => "foofoofoo"

def bar baz: "foo" * 3; baz end
bar # => "foofoofoo"

However, when I try to evaluate a local variable/method under a certain scope in the default value description as follows, the local variable/method is evaluated under lexical scope:
MAIN = TOPLEVEL_BINDING.eval('self')
foo = 3

def bar baz = MAIN.instance_eval{foo}; end
bar # => undefined local variable or method `foo' for main:Object

def bar baz: MAIN.instance_eval{foo}; end
bar # => undefined local variable or method `foo' for main:Object

*

*Why is foo above not evaluated within MAIN scope and is evaluated in lexical scope?

*This seems there is some limitation to what Ruby expressions can be evaluated in the default value description. What exactly can be put there?

A: foo is local variable for main. Your attempt to access local variable from outside might be shorten to:
▶ MAIN = TOPLEVEL_BINDING.eval('self')
▶ foo = 3
▶ MAIN.foo
#⇒ NoMethodError: undefined method `foo' for main:Object

The analogue of this code in less tangled manner is:
class A
  foo = 5
end

class B
  def a_foo
    A.new.foo
  end
end

▶ B.new.a_foo
#⇒ NoMethodError: undefined method `foo' for #<A:0x00000002293bd0>

If you want to provide access from the universe to your local variable you are to implement getter:
def MAIN.foo ; 5 ; end
def bar baz = MAIN.instance_eval{foo}; baz; end

▶ bar
#⇒ 5

Hope it helps.


text:
Q: Using *(pointer+i) versus pointer[i] Can somebody please explain this code
// Option 1
int **p = new Point*[2];
p[1] = new Point;
p[1]->x = p[1]->x = 1;

// Option 2
int **p = new Point*[2];
*(p+1) = new Point;
(*(p+1))->x = (*(p+1))->x = 1;

Isn't both options the same? Why when I create the variable using option 2, I cant write its value with option 1 (I got random numbers (address numbers?))? Is there any difference?

A: You have a typo: (*(p+1))->x = (*(p+1))->x = 1; - should be y in the second term


text:
Q: Object design for REST result, including error handling I'm not sure what the best design is for returning an object that can both return a result for successful REST calls, or failed REST calls.
I'm currently thinking of something like this (written in Java):
public class ResultDTO<T> {
    public enum Status {
        SUCCESS, FAILURE
    }

    private Status status;
    private String message;
    private T result;
}

So it can be used like:
return new ResultDTO<User>(Status.SUCCESS, "", new User("john doe")); // Correct usage of API
return new ResultDTO<User>(Status.FAILURE, "Invalid e-mail", null); // Incorrect usage of API

But I'm not quite sure about this design. Does anyone have any experience with designing Objects for REST API's and can recommend me something?

A: It's difficult to provide a definitive answer to this because REST is a "style" of building applications and because you don't provide enough context of how your application works to produce the results, but nevertheless I'll try to offer some advice.
It's good that you want to use a homogeneous way to present all your results, be them of a success or error status, but you should not represent both with the same class. Take this code for example:
new ResultDTO<User>(Status.SUCCESS, "", new User("john doe"));
new ResultDTO<User>(Status.FAILURE, "Invalid e-mail", null);

See anything wrong with it? The only useful information here is new User("john doe") and "Invalid e-mail". The rest is noise. Having an empty string for the message in the successful case, and a null result in the error case are a giveaway that the results should be exclusive, something like:
new SuccesfulResult<User>(new User("john doe"));
new ErrorResult("Invalid e-mail");

Your code might turn into this to reflect the new semantic:
public abstract class AbstractResult {
    public enum Status {
        SUCCESS, FAILURE
    }

    private final Status status;

    protected AbstractResult(Status status) {
        this.status = status;
    }

    private final Status getStatus() {
        return this.status;
    }
}

public class SuccesfulResult<T> extends AbstractResult {
    private T result;

    public SuccesfulResult(T result) {
        super(AbstractResult.Status.SUCCESS);
        this.result = result;
    }

    // ...
}

public class ErrorResult extends AbstractResult {
    private final String message;

    public ErrorResult(String message) {
        super(AbstractResult.Status.FAILURE);
        this.message = message;
    }

    // ...
}

It's a lot more code to write, but you write it once. You will call it a lot more times in your application and it beats having to always type "" and null all over the place.
Now, about the comment on your question. Don't make this part of your domain. The T result is part of your domain. The rest is just metadata that accompanies the result so that the client knows if it's a successful call or a failed one. Your application should work with T (whatever that represents) and only add the result metadata when marshalling the response for the client. Frameworks can help you there, especially for the error handling. In your application code you might throw an exception then let the framework catch it and transform it in a new ErrorResult(...) based on a mapping you provide.
Above are my observations regarding the code you posted. For other information try searching for "good REST API design" and "best practices for handling errors in rest APIs". You'll find more information there that you might not have though of, and it might be more useful to you than any solution you receive on this post.


text:
Q: Flexibility with integration limits I have an EE background and I am trying to integrate the equation for the voltage across an inductor. I am just playing around with the limits and want to check if all 3 of these equations below are correct or if I have made a mistake:

Are 1, [2] and [3] all correct?

A: For any $a,b$, we have by the fundamental theorem of calculus that
$$
i(a) - i(b) = \frac{1}{L}\int_a^b V_L(\tau)d\tau.
$$
Let's compare this to your equations.

*

*This is only true if $i(-\infty) = \lim_{t\rightarrow -\infty}i(t) = 0$.

*This is never true even with the $1/L$ correction, as $V_L(0)/L$ has the wrong units. That term should instead be $i(0)$.

*This is always true.


text:
Q: MySQL Server does not start on Raspberry Pi I am working with a Raspberry Pi running Debian Linux. I am trying to install  MySQL Server and run it and here is what I have done:
# sudo apt-get install mysql-server mysql-client php5-mysql

No errors. Now I rebooted the system, then when starting up I get:
[FAIL] startpar: service(s) returned failure: mysql ... failed!

And I tried
# sudo service mysql start
mysql: unrecognized service

Lastly I tried
# /etc/init.d; mysql
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

I find that I have no folder named mysqld in /var/run. But I read that the sock file here shall be created the first time the server is created?
Do you have any suggestions?

A: Looks like your mysql server is not started. I usually run the stop command and then start it again:
mysqld stop
mysql.server start

Same error, and this works for me.

A: /etc/init.d; mysql

The usual way to start the mysql server would be
/etc/init.d/mysqld start

The reason it's failing should be written to the logs (usually /var/log/mysqld.log) - what do the logs say?

A: look into /var/log/mysql/error.log


text:
Q: How to display the previous result if user click refresh the page in jquery? I am working Image gallery website. Image search page will work like google image search. On scroll it shows next results using jquery. How to display the same result if user refresh the page?
Thanks, Sham

A: You can use query string to get the user search parameters or you can use the search text field to show the result on refresh of page.

A: Using query parameter will help only to show the page from the beginning after refresh, But what i would do is,
There are two options,

*

*Using cookies

*Using HTML5 local Storage or session storage
I prefer the second one,
But in both way you can cache the current search data, therefore can be used later. This will help you to display different page of results, where the query parameter remains same.


text:
Q: Android - Test mobile app in tablet I am creating an android application which is for both android mobile phones and tablet. I am having only android tablet to test the app(We can test using emulators with different screen sizes). Is it possible to test android mobile apps on android tablet but not in full screen? real size in the middle of the screen(Like IOS: In Ipad, we can test the iphone app).

A: Is it possible to test android mobile apps on android tablet but not in full screen?

No , this is not possible.

*

*To test on android phone you need to test on actual device or an emulator only

*In the graphics layout of the xml there is a option to select device size ,the default is Nexus one.Using this option you can preview the layout for different screen sizes .

Hope it helps.


text:
Q: How to change number of digits separately for columns in renderTable in R Shiny I have a table to display in R Shiny, using renderTable, but I would like to change the number of digits only for the second column into 3 digits, and the first and third column into 0 digits. How is this possible?
   output$Table_X <- renderTable({

        outputData <- loadFunc()
        outputData$x
      },
      digits = 3)

In the line with outputData <- loadFunc(), the list outputData is created, with a dataframe in it called 'X'. This contains three columns with numeric data. Now it the rendertable displays the data as following:
a               b                 c
0.000           1.123             10.000
2.000           2.345             12.000
4.000           5.789             15.000

I would like to have only column 'b' with 3 number of digits, and column 'a' and 'c'with zero number of digits.

A: One solution is to use formatRound in package DT. This requires to use dataTableOutput and renderDataTable, and to put your data in datatable():
library(shiny)
library(tibble)
library(DT)

data <- tibble(
  x = c(1.889, 1.289),
  y = c(3.44898, 4.48349),
  z = c(3.4343, 3.1)
)

ui <- fluidPage(
  dataTableOutput("table")
)

server <- function(input, output, session) {

  output$table <- renderDataTable({
    datatable(data) %>%
      formatRound(columns = 2, digits = 3) %>%
      formatRound(columns = c(1, 3), digits = 0)
  })

}

shinyApp(ui, server)

Edit: If you don't want to use dataTableOutput and renderDataTable, you can replace the function formatRound by formatC (in base R), but it is a bit more annoying:
library(shiny)
library(tibble)

data <- tibble(
  x = c(1.889, 1.289),
  y = c(3.44898, 4.48349),
  z = c(3.4343, 3.1)
)

ui <- fluidPage(
  tableOutput("table")
)

server <- function(input, output, session) {

  output$table <- renderTable({
    data$x <- formatC(data$x, digits = 0)
    data$z <- formatC(data$z, digits = 0)
    data$y <- formatC(data$y, digits = 3)
    data
  })

}

shinyApp(ui, server)


text:
Q: Making click events into touch events when run on tablets I have made a spelling game for primary school children. I want to make the click events in my game touch events so it can be compatible on tablets. Is there a way I can put them alongside my click events so that one program is compatible to both, or will I have to make a tablet version?
Here I have the code for one of my click events as an example
$('.drag').on('click', function(e) {
e.preventDefault();
if (animation) return;
animation = true;
setTimeout(function() {
    animation = false;
}, 700);
$(".minibutton").css('visibility', 'hidden');
$('.next').css('visibility', 'hidden');

var target = $('.drop-box.spellword:not(.occupied):first');
var targetPos = target.position();
var currentPos = $(this).offset();
var b = $(this);

if (target.length) {
    target.addClass("occupied");
    b.clone().addClass(
    b.data("letter") == target.data("letter") ? "wordglow3" : "wordglow").appendTo("table").css({
        background: "transparent",
        position: "absolute",
        top: currentPos.top,
        left: currentPos.left
    }).animate({
        top: targetPos.top,
        left: targetPos.left
    }, "slow", function() {
        $(this).css({
            top: 0,
            left: 0
        }).appendTo(target);

        var spellWord = $('.drop-box.spellword');
        if (!spellWord.filter(':not(.occupied)').length) {
            var wordIsCorrect = 0;
            spellWord.each(function() {
                if ($(this).data("letter") == $(this).find("div").data("letter")) {
                    wordIsCorrect++;
                }
            });

            if (spellWord.length == wordIsCorrect) {

                spellWord.addClass('wordglow2');
                $(right).val('Right!');
                $(right).show();
                success.play();
                $(wrong).hide();
                score.right++;
                score.attempts++;

                if (score.right == 3) {

                    $('.answers').css('visibility', 'visible');
                    $('.answers').html("Well done! </br> You correctly spelt " + score.right + ". </br> Keep it up.").show();
                    $('table').fadeOut(3000);
                    $('.right').hide();
                    $('.box-style2').hide();
                    $('.box-style').hide();
                    $('.picstyle').hide();
                    $('.play').hide();
                    $('.minibutton2').hide();
                    $("#mysoundclip").attr('src', listOfWords[rndWord].audio);
                    audio.stop();
                    $("#mypic").attr('src', listOfWords[rndWord].pic);
                    pic.hide();

                }

                setTimeout(function() {
                    jQuery('.minibutton').trigger('click');
                }, 1500);

                setTimeout(function() {
                    jQuery(right).hide();
                }, 1500);

            } else {

                //spellWord.addClass("wordglow4").css('color', 'transparent');
                $(wrong).val('Wrong!');
                $(wrong).show();
                failure.play();
                $(right).hide();
                score.wrong++;
                score.attempts++;

                if (score.wrong == 3) {

                    $(".minibutton").css('visibility', 'visible');
                    $('.next').css('visibility', 'visible');

                }

                $('.drop-box.spellword').animate({
                    'opacity': 1
                }, 1500, function() {
                    $(this).removeClass('wordglow4').removeClass('occupied').html('')
                });

                setTimeout(function() {
                    jQuery(wrong).hide();
                }, 1500);

            }
        }
    });

}

});

Can someone point me in the right direction as I have never done this before.
Here is a fiddle to help (Sound warning!!) http://jsfiddle.net/smilburn/7Y7A5/8/

A: You could add the tap event and include jQuery Mobile in your mobile version.
$('.drag').on('click tap', function(e) {...}


text:
Q: Retrieve the session value and pass as required parameter of api Ajax I want to retrieve the session value and pass that value as required parameter for the api. My session is $_SESSION['name']. And these are my codes
var uniqueName = **Here I want to retrieve session value**;
  if(**Then I want to check if uniqueName is null or undefined**) {
    $.ajax({
      url: 'http://api/check.php',
      type: 'GET',
      crossDomain: true,
      dataType:'json',
      data: {
        name: uniqueName
      },
      success: function(result) {
        alert("Success");
      },
      error: function(result) {
        alert("Error");
      }
    });
  }

  return false;
});

A: You can put PHP directly into your JS, like
var uniqueName = '<?php echo isset($_SESSION['name']) ? $_SESSION['name'] : '' ?>';

In JS will be name or empty string.

If you haven't this JS in PHP file, you can pass a session using AJAX call, or assign to any HTML element.
<input type="hidden" id="session_var" value="<?php echo isset($_SESSION['name']) ? $_SESSION['name'] : '' ?>">

And in JS file then:
var name = document.getElementById('session_var'); // name from session or empty string.

A: var uniqueName = "<?php echo $_SESSION['name'] ?>";
  if(uniqueName) {
    $.ajax({
      url: 'http://api/check.php',
      type: 'GET',
      crossDomain: true,
      dataType:'json',
      data: {
        name: uniqueName
      },
      success: function(result) {
        alert("Success");
      },
      error: function(result) {
        alert("Error");
      }
    });
  }

  return false;
});

References:
Is there a standard function to check for null, undefined, or blank variables in JavaScript?
http://www.dyn-web.com/tutorials/php-js/scalar.php


text:
Q: Enviar datos de formulario oculto cuando escribo los datos de un usuario en wordpress para registrarse, deseo que al pulsar el botón "Registrar o Enviar" este acceda al sitio, es decir que quede logueado de manera automática. (mediante código ya he logrado activarlo sin necesidad de confirmar por email)
Se me ocurre tener un formulario oculto en el mismo sitio que llene los datos de los inputs con los valores del usuario recién registrado, y acceda de inmediato. Para esto pienso que al pulsar el botón Registrar realice el envío de los dos formularios, uno que registra y otro que loguee al usuario.
entiendo puedo usar inputs ocultos, pero que código puedo usar para que al pulsar el botón Registrar, también se active el botón Login.
de antemano muchas gracias.
<input id="prodId" name="prodId" type="hidden" value="xm234jq">

A: perfecto Javi PS no conocía esa forma
indagando por la web me encontré con esta función propia de Wordpress
y como bien decían los demás compañeros estaba complicandome la vida
aqui la funcion
wp_signon( array $credentials = array(), string|bool $secure_cookie = '' )
aquí un ejemplo de la misma página, voy a probar ambas.
/**
 * Perform automatic login.
 */
function wpdocs_custom_login() {
    $creds = array(
        'user_login'    => 'example',
        'user_password' => 'plaintextpw',
        'remember'      => true
    );

    $user = wp_signon( $creds, false );

    if ( is_wp_error( $user ) ) {
        echo $user->get_error_message();
    }
}

// Run before the headers and cookies are sent.
add_action( 'after_setup_theme', 'wpdocs_custom_login' );


text:
Q: How to handle many "Then" step definitions? Consider a scenario with multiple "Then" step definitions meant to be used as assertions against a response payload:
  ...
  When a response is received
  Then the response should have an element "foo" with the content "bar"
  And the response should contain 1 "foobar" element
  And the response should have an element "rab" with the content "oof"
  ...

What is the intended way for Citrus to handle an unknown amount of validations? Can you define several validators before calling receive()? Can this be handled with a validationCallback() and minimal Gherkin rewrite?
The current implementation uses validationCallback() to store the payload as an instance variable, then validate against the variable. However, it would be much better to leverage the power of Citrus.

A: You could use named messages in a Given section or even in a scenario background:
Given message fooResponse
    And <fooResponse> payload is "Hi my name is Foo!"
    And <fooResponse> header operation is "sayHello"
    ...

You can use the named message in a scenario like this:
Scenario: Send and receive
  When <client> sends message <fooRequest>
  Then <client> should receive message <fooResponse>

The validation of all elements is done automatically as the Citrus validator compares received and expected message contents.
Also you could extract the message creation to a Java POJO message creator:
Background:
  Given message creator com.consol.citrus.FooMessageCreator

public class FooMessageCreator {
    @MessageCreator("fooResponse")
    public Message createEchoResponse() {
        return new DefaultMessage("Hi my name is Foo!")
                    .setHeader("operation", "sayHello");
    }
}

In case you need to deal with large XML/Json payloads you can also load that payloads from external file template resource. Instead of setting each element in a separate explicit Then statement you should go for template based compare validation that can also use @ignore@ sections for ignoring parts of the message that are not relevant for validation.


text:
Q: Two Facts - Two different grains I'm just getting back into SSAS after inheriting an existing cube and I'm not sure how to proceed with this scenario:
Budget Fact:

*

*Product

*Customer

*Time

Promo Fact:
 - Product
 - Customer
 - Promotion
 - Time
I'm looking to do a report combining measures from the Budget Measure Group and Promo Measure Group.
Essentially I'm trying to get a list of Customers and Products where the Promotion.Discount Value > .4.
This means I have to pull the Promotion Dimension into my dataset and that's where things start getting hairy.
A report on Customer, Product, Promo ID, and Promo.Amount is 1565 records.
A report on Customer, Product and Budget.Amount is 31 records.
A report on Customer, Product, Promo ID, Promo.Amount and Budget.Amount is 179,878 records!
Really what I'm trying to achieve (from a SQL point of view) is to return a list of Customers and Products where the Promo Discount Value is > .4 then link that to another dataset on Customer ID and Product ID to get the budget value.
I've spent a fair bit of time reading about the Dimension Usage facility etc and nothing seems to be helping. Any advice?!

A: If the cube is already in place and correctly constructed, then you shouldn't have to change anything in Dimension Usage. But just to make sure, when you look at that page, you should see an attribute name appear in each intersection of dimension with the corresponding measure group. Based on your description, only the intersection of Promotion and Budget should appear as a gray (empty) cell.
It sounds like the problem is how to write the appropriate MDX query. You should be able to produce the list you want with a query similar to this (changing actual names of dimensions and measures as applicable):
with
set CustomerProductList as
Filter((Customer.Customer.Customer.Members, Product.Product.Product.Members, Promotion.Promotion.Promotion.Members),
Val(Promotion.Promotion.CurrentMember.Name) > .4)

select
{Measures.BudgetAmount, Measures.PromotionAmount} on columns,
CustomerProductList on rows
from YourCube


text:
Q: "fatal: cloning the git-repo repository failed, will remove '.repo/repo' " upon running repo init <...> I've been running into this issue every time I run a repo init -u https://source.codeaurora.org/quic/le/le/manifest.git -b [branch] -m [manifest]  command.
  >>
  >> repo 1.12.16
  stderr:
  >> gpg: Signature made Thu 01 May 2014 01:34:18 PM PDT using RSA key ID 692B382C
  >> gpg: Can't check signature: public key not found
  >> error: could not verify the tag 'v1.12.16'
fatal: cloning the git-repo repository failed, will remove '.repo/repo'

Then I installed the repo package via sudo apt-get install repo which seems to install the repo in /usr/bin dir so upon running $/usr/bin/repo init -u https://source.codeaurora.org/quic/le/le/manifest.git -b [branch] -m [manifest] , I now see:
repo: error: "git" failed with exit status 1
  stderr:
  >> gpg: Signature made Thu 22 June 2012 01:14:15 PM PDT using RSA key ID 692B382C
  >> gpg: Can't check signature: public key not found
  >> error: could not verify the tag 'v1.12.16'
fatal: cloning the git-repo repository failed, will remove '.repo/repo'

I tried this approach to no avail.


text:
Q: Check with VBA if an element exists on the page I have 15,000 products and I need to know if they're X, Y or Z. The code below checks on amazon to see if a product is a type of XYZ.
God help me, it actually works. The only exception is when it searches a product no longer sold by Amazon. Then the element ID that I'm looking for which contains the product description that I'm searching doesn't exist on the page, and the code breaks on line
text = document.getelementbyID("result_0").innertext

with the error "Object variable or With block variable not set".
How do I check if the element exists before proceeding with the rest of the code?
Thanks!
Sam
Sub LetsAutomateIE()

Dim barcode As String
Dim rowe As Integer
Dim document As HTMLDocument
Set ie = CreateObject("InternetExplorer.Application")
Dim Element As HTMLDivElement
Dim text As String
Dim pos As Integer

rowe = 2

While Not IsEmpty(Cells(rowe, 2))

barcode = Cells(rowe, "B").Value

With ie
.Visible = False
.navigate2 "https://www.amazon.co.uk/s/ref=nb_sb_noss_1?url=search-
alias%3Daps&field-keywords=" & barcode
Do Until ie.readyState = 4
Loop
End With

Set document = ie.document

text = document.getElementById("result_0").innerText

If InStr(text, "X") Or InStr(text, "Y") Or InStr(text,
"Z") <> 0 Then pos = 1

If pos <> 0 Then Cells(rowe, 4) = "Y" Else Cells(rowe, 4) = "N"

rowe = rowe + 1

Wend

Set ie = Nothing

End Sub

A: Posting an alternative in case top answer is not working for other people:
I was still getting run time error '91' after checking with IsObject()
IsObject() works fine if ObjIE is an InternetExplorer object but I was using an InternetExplorerMedium object and had to validate with this instead since it was returning a nothing object:
If Not objIE.document.getElementById("e164NumberMask") Is Nothing Then
    'do true stuff
Else
    'do false stuff
End If

A: Ryan's answer is the correct.
set Element = document.getelementbyID("result_0")

A: Thanks for the solution. I used as follows,
If IsObject(objIE.document.getElementById("e164NumberMask")) Then
    'do true stuff
Else
    'do false stuff
End If

A: You can try something like this.
Function objectHandler(objID)

Dim TestObj As Object

On Error GoTo Handler:

Set TestObj = ObjIE.document.getElementById(objID)
objectHandler = True
Exit Function

Handler:
objectHandler = False

End Function


text:
Q: WPF ComboBox in grid - how to specify search path in viewmodel We have a DataGridTemplateColumn that uses a ComboBox in the DataTemplate:
                <DataGridTemplateColumn Header="Things">
                <DataGridTemplateColumn.CellTemplate>
                    <DataTemplate>
                        <ComboBox
                          IsTextSearchEnabled="True"
                          TextSearch.TextPath="Name"
                          ItemsSource="{Binding Things}">
                            <ComboBox.ItemTemplate>
                                <DataTemplate>
                                    <StackPanel Orientation="Horizontal">
                                        <TextBlock Text="{Binding Index}" Margin="0,0,12,0" />
                                        <TextBlock Text="{Binding Name}" />
                                    </StackPanel>
                                </DataTemplate>
                            </ComboBox.ItemTemplate>
                        </ComboBox>
                    </DataTemplate>
                </DataGridTemplateColumn.CellTemplate>
            </DataGridTemplateColumn>

We need to let the user specify the search path for the ComboBox.
 TextSearch.TextPath="{Binding SearchPath}"

How do I specify that this binding refers to the top-level viewmodel, not the current item from the Things collection (which Name and Index refer to)?
Thanks for any insights --