text:
Q: Pyspark: how to select a subset of a dataset and save it? I have a dataset that I ma trying to manage with pyspark and for which I would like to select a subset. Let me say that I am quite new to spark, scala and pyspark in general.
This is what I am doing.
from pyspark.sql import SparkSession
from pyspark.sql.types import *
import pyspark.sql.functions as fs

spark  = SparkSession.builder\
                  .master("local")\
                  .enableHiveSupport()\
                  .getOrCreate()

spark.conf.set("spark.executor.memory", '16g')
spark.conf.set('spark.executor.cores', '16')
spark.conf.set("spark.shuffle.compress", 'True')
spark.conf.set("spark.python.worker.memory", '16g')
spark.conf.set("spark.default.parallelism", '10')

sc = spark.sparkContext

%%time
f = "myData"
schema = StructType([
    StructField("ID", StringType(), True),
    StructField("Code", StringType(), True),
    StructField("bool", IntegerType(), True),
    StructField("lat", FloatType(), True),
    StructField("lon", FloatType(), True),
    StructField("v1", FloatType(), True),
    StructField("v2", IntegerType(), True),
    StructField("v3", FloatType(), True)])

df = spark.read.format("csv").schema(schema).load(f)
df.show()

+--------------------+----+----+---------+----------+---------+----------+---------+
|                  ID|Code|bool|      lat|       lon|       v1|        v2|       v3|
+--------------------+----+----+---------+----------+---------+----------+---------+
|5ac52674ffff34c98...|IDFA|   1|42.377167| -71.06994|17.422535|1525319638|36.853622|
|5ac52674ffff34c98...|IDFA|   1| 42.37747|-71.069824|17.683573|1525319639|36.853622|
|5ac52674ffff34c98...|IDFA|   1| 42.37757| -71.06942|22.287935|1525319640|36.853622|
|5ac52674ffff34c98...|IDFA|   1| 42.37761| -71.06943|19.110023|1525319641|36.853622|
|5ac52674ffff34c98...|IDFA|   1|42.377243| -71.06952|18.904774|1525319642|36.853622|
|5ac52674ffff34c98...|IDFA|   1|42.378254| -71.06948|20.772903|1525319643|36.853622|
|5ac52674ffff34c98...|IDFA|   1| 42.37801| -71.06983|18.084948|1525319644|36.853622|
|5ac52674ffff34c98...|IDFA|   1|42.378693| -71.07033| 15.64326|1525319645|36.853622|
|5ac52674ffff34c98...|IDFA|   1|42.378723|-71.070335|21.093477|1525319646|36.853622|
|5ac52674ffff34c98...|IDFA|   1| 42.37868| -71.07034|21.851894|1525319647|36.853622|
|5ac52674ffff34c98...|IDFA|   1|42.378716| -71.07029|20.583202|1525319648|36.853622|
|5ac52674ffff34c98...|IDFA|   1| 42.37872| -71.07067|19.738768|1525319649|36.853622|
|5ac52674ffff34c98...|IDFA|   1|42.379112| -71.07097|20.480911|1525319650|36.853622|
|5ac52674ffff34c98...|IDFA|   1| 42.37952|  -71.0708|20.526752|1525319651| 44.93808|
|5ac52674ffff34c98...|IDFA|   1| 42.37902| -71.07056|20.534052|1525319652| 44.93808|
|5ac52674ffff34c98...|IDFA|   1|42.380203|  -71.0709|19.921381|1525319653| 44.93808|
|5ac52674ffff34c98...|IDFA|   1| 42.37968|-71.071144| 20.12599|1525319654| 44.93808|
|5ac52674ffff34c98...|IDFA|   1|42.379696| -71.07114|18.760069|1525319655| 36.77853|
|5ac52674ffff34c98...|IDFA|   1| 42.38011| -71.07123|19.155525|1525319656| 36.77853|
|5ac52674ffff34c98...|IDFA|   1| 42.38022|  -71.0712|16.978994|1525319657| 36.77853|
+--------------------+----+----+---------+----------+---------+----------+---------+
only showing top 20 rows

CPU times: user 4 ms, sys: 4 ms, total: 8 ms
Wall time: 3.44 s

Let say that I would like to select the information of the first ID.
%%time
id0 = df.first().ID  ## First ID
tmp = df.filter( (df['ID'] == id0) )

CPU times: user 0 ns, sys: 8 ms, total: 8 ms
Wall time: 514 ms

%%time
tmp.count()
CPU times: user 8 ms, sys: 4 ms, total: 12 ms
Wall time: 1min 25s
Out[5]:
3299

The output is a pyspark.sql.dataframe.DataFrame with 3299 rows. The first question is why is taking so long time to count 3299 rows?
The second question is: how can I save it or transform it as a normal dataframe? Is it normal that is taking so long to convert it to a normal dataframe?
%%time
result_pdf = tmp.select("*").toPandas()
CPU times: user 60 ms, sys: 12 ms, total: 72 ms
Wall time: 3min 33s

A: Well, both your questions are related to the RAM.

*

*The fact that Spark doesn't load data in RAM (contrary to Pandas) explain the slowness. The thing is 3200 row is not big enough to use a Spark I would say.
If you can use Pandas, better using it because it will be faster.
If you want to scale to multiple millions / billions and you can't load your data in RAM (because Pandas is really greedy : for 1GB dataset, you will probably need 10GB of RAM) so you can Out Of Memory your computer fast.
So, if you got multiple GB or TB (or even PB), Spark is  for you. If you got less than that and can fit your data in RAM, then use Pandas
So the speed scaling is not enough fast for you to see the difference but Spark will probably be as fast for 3K data and 3M, because it splits execution, distribute the calculation, then gather it again etc. Contrary to Pandas that does all the calculation in RAM.

*The conversion speed (toPandas()) is also slow, because you're loading all the data into memory (RAM) so the transfer. It's really like loading your data with Pandas and then filtering.

If you want to save it, the best way is :
data.write.parquet("data.parquet")

or
data.write.csv("data.csv")

Further documentation :
https://spark.apache.org/docs/latest/sql-data-sources-load-save-functions.html


text:
Q: Function to read data from a file into heap memory I am writing a function that reads the number of grades in a text file and stores them into heap memory. It writes the count to location pointed to by ptr_cnt and returns the address of the first byte stored into said heap memory.
I have written this and the count works just fine. However the data does not seem to be getting written into the the heap. When I print out the first heap, I get 0.000.
double *read_data(char const *file_name, int *ptr_cnt)
{
    FILE *read = fopen(file_name, "r");
    if (read == NULL)
    {
        return NULL;
    }
    int count = 0;
    char ch;
    while ((ch = fgetc(read)) != EOF)
    {
        if (ch == ' ' || ch == '\t' || ch == '\n' || ch == '\0')
        {
            count++;
        }
    }
    *ptr_cnt = count;

    double *heap;
    heap = (double *)malloc(sizeof(double) * count);
    double num = 0.0;
    int i;
    for (i = 0; fscanf(read, "%lf", &num) != EOF; i++)
    {
        *(heap + i) = num;
    }
    fclose(read);
    return heap;
    free(heap);
}

redpajama_set_name: RedPajamaStackExchange


text:
Q: why my text field show ??? in sql server? evrything is ok but my text is ???
       select sum(mande) jameMandeha,
   [عنوان]=
  CASE
       WHEN [ردیف] =1 THEN TRY_CAST(('سپرده های قرض الحسنه جاری - ریال') AS NVARCHAR(MAX))
       WHEN [ردیف] =2 THEN  CAST(('سپرده های قرض الحسنه جاری - ارز') AS NVARCHAR(MAX))
       WHEN [ردیف] =3 THEN  CAST(('انواع چک های فروخته شده') AS NVARCHAR(MAX))
       WHEN [ردیف] =7 THEN  CAST(('وجوه اداره شده مصرف نشده - ریال') AS NVARCHAR(MAX))
       WHEN [ردیف] =9 THEN  CAST(('بستانکاران موقت - ریال') AS NVARCHAR(MAX))
       WHEN [ردیف] =10 THEN CAST(('بستانکاران موقت - ارز') AS NVARCHAR(MAX))
       WHEN [ردیف] =11 THEN CAST(('[مانده مطالبه نشده - ریال]') AS NVARCHAR(MAX))
  END,
  [ردیف],regdate from #tmp_res group by [ردیف],regdate order by [ردیف]

but have this result
3833456009228   ????? ??? ??? ?????? ???? - ????    1   1397/06/31
150763478110    ????? ??? ??? ?????? ???? - ???     2   1397/06/31
366674660068    ????? ?? ??? ?????? ???             3   1397/06/31
2103192861      ???? ????? ??? ???? ???? - ????     7   1397/06/31
2547718000      ?????????? ???? - ????              9   1397/11/29
1107433985359   ?????????? ???? - ????              9   1397/06/31
2979163         ?????????? ???? - ???               10  1397/06/31
12336285349     ????? ?????? ???? - ????            11  1397/06/31

thanks for read my problem,please help me


text:
Q: Is it possible to reload a Ruby script without restarting it? I am writing an IRC service in Ruby.
Here is some code:
#!/usr/local/bin/ruby

require "socket"

$SAFE=1

class IRC
    def initialize(server, port)
        @server = server
        @port = port
    end
end

At the moment when I change the code I kill the PID and restart it but I was wondering how to implement a sort-of 'rehash' to reload the code without restarting the process (causing a disconnect/connect).


text:
Q: Cayley graph of Rubik's cube group
(a) I would like to know whether there is a group theoretic approach for calculating the diameter of the Cayley graph of Rubik's Cube group.

I know it's been proved that the above diameter is $20$ but the approach uses brute force.
Also I wonder whether there is a "nice" presentation of this group (it is a finite group so the relations between the elements of this group come from the multiplication table, but (b) is there a systematic way of writing down these relations?)

What about a $2\times2\times2$ Rubik's Cube? (Is it still hard to examine the Cayley graph?)

EDIT
Since there has been no full answer to my initial questions I'd like to ask for something else too:

Is there an "nice" way to describe a maximal tree of the Cayley graph of the $3\times3\times3$ Rubik's Cube?


text:
Q: Why markdown syntax requires two spaces before \n to make a paragraph out of it? I use this library to create markdown text editor in a webapp.
All formatting is intuitive, but the line breaks. I don't get, why i have to add double space and then \n to create line break? Is there other markdown => html js parser, which doesn't require this?

A: It doesn't require two spaces before a newline to make paragraphs. Paragraphs in Markdown are defined by two consecutive newlines (i.e. there must be a blank line in between the paragraphs). The "two spaces" syntax is for line breaks, i.e.
 tags.
The reason for the two space rule for line breaks is shown in the documentation (bold mine):

A paragraph is simply one or more consecutive lines of text, separated by one or more blank lines. (A blank line is any line that looks like a blank line — a line containing nothing but spaces or tabs is considered blank.) Normal paragraphs should not be indented with spaces or tabs.
The implication of the "one or more consecutive lines of text" rule is that Markdown supports "hard-wrapped" text paragraphs. This differs significantly from most other text-to-HTML formatters (including Movable Type's "Convert Line Breaks" option) which translate every line break character in a paragraph into a
 tag.
When you do want to insert a
 break tag using Markdown, you end a line with two or more spaces, then type return.
Yes, this takes a tad more effort to create a
, but a simplistic "every line break is a
" rule wouldn't work for Markdown. Markdown's email-style blockquoting and multi-paragraph list items work best — and look better — when you format them with hard breaks.


text:
Q: Find code that could be reached by a specific function I have a hard time formulating this question. The reason I'm asking is that I want to convert some C++ code with emscripten to java script code, but I don't think I need to convert the whole code base.
Is it possible in C++ to find all the code that a particular function could reach into when executed? Then I would know which part of the code I need to convert and which one I can just ignore.

A: It is called "call hierarchy" as Eugene said. You can use automatic documentation tools to get this information.
I strongly recommend you to try doxygen because it is really easy to use:
http://www.doxygen.nl/


text:
Q: Add value from @string to my array.xml I have my array.xml to fill preferences value.
<string-array name="language">
    <item>English</item>
    <item>German</item>
    <item>Russian</item>
    <item>Italian</item>
</string-array>

To translate the items I put them in various string.xml file, how can I add the values from  @string/english, @string/german etc.?

A: Have you tried something like this:
res/values/array.xml
res/values-fr/array.xml
res/values-ja/array.xml

And so on...
If your issue is that you want to substitute the <item> value dynamically, you might have to do this in code. Check out this post: dynamic parameters in strings

A: You can just create string arrays in string.xml and other string.xml(ru) etc.
See below code snippet :-
string.xml
<!-- Language array -->
<string-array name="languages">
    <item>English</item>
    <item>Russian</item>
</string-array>

string.xml(ru)
<!-- Language array -->
<string-array name="languages">
    <item>английский</item>
    <item>русский</item>
</string-array>

A: do this...
<string name="jan">January</string>
<string name="fev">February</string>
<string name="mar">March</string>

<string-array name="year">
    <item name="jan">@string/jan</item>
    <item name="fev">@string/fev</item>
    <item name="mar">@string/mar</item>
</string-array>


text:
Q: rails - passing method name to helper function docs say that options_from_collection_for_select should be used following way:
options_from_collection_for_select(collection, value_method, text_method, selected = nil)

so, in my case for example
options_from_collection_for_select(@messages,'id','title')

but i need to put more information to title, so what i tried to do was:
class Message < ActiveRecord::Base

 def proper_title
  self.name+", updated at "+self.updated_at
 end

end

and it works, but thing is i need strings internationalized and it's a bit more difficult with models than with controllers.
now do i have to do model internationalization in this case or is it possible to get around somehow? thanks

A: You can still call I18n.translate() in the model. It will give you the same result as t() helper
# Message.rb
def proper_title
  I18n.translate("message.proper_title", :name => self.name, :updated_at => self.updated_at)
end

# en.yml
en:
  message:
    proper_title: "{{name}}, updated at {{updated_at}}"

# view
options_from_collection_for_select(@messages,'id','proper_title')


text:
Q: WPF giving maximize,minimize,resize control to child window Hosting a child window from different process into my WPF application.
Parent window has WindowStyle = "None".
I want to give control for resize & move to the child window i.e. parent window will maximize, minimize, close, move, resize corresponding to the child window operations.

A: For each control you can set under Properties

Layout
  Anchor  -> Top, Bottom, Left, Right

this will fit your control to  the child : Windows Form , Tabs, etc .


text:
Q: unsupported operand type +: built in function or method and str I understand what the error is saying, I just don't understand why a function used on a string won't concatenate with a string or how to go about solving that issue. The code seems straight forward to me.
Here is the simplest code I made to test this.
word = word[0].upper() + word[1:].lower

A: You forgot the () after .lower:
word = word[0].upper() + word[1:].lower()