text:
Q: How to determine saturation in DC smoothing choke with capacitor, and how to simulate in LTspice? This question addresses some of the issues with a choke made from a microwave oven transformer for producing smooth DC from a phase-fired SCR controller, as discussed in this question.
The choke measured 3 mH and has 10000 μF capacitance on a nominal 22 Ω load, which will see an initial voltage of about 220 VDC and then lowered to about 75 VDC for continuous operation.
I ran the following simulations using an ideal inductor model, showing waveforms for current and voltage at 50% phase modulation (90°) for 220 V, and 20% modulation (144 °) for about 75 VDC. The current through the choke was 19.2 A RMS at 50% and 7.7 A RMS at 20%. The power dissipated in the choke, based on 50 mΩ winding of 30 turns (30 ft) of #12 AWG wire, was 7.7 W at 20% and 18.8 W at 50%. However, the actual winding may have been as much as 100 ft, so actual wattage could be three times the above figures.
Simulation at 50%:

Simulation at 20%:

Then, to simulate what might happen if the choke saturated, I applied an effective short across the choke using a low RDS(on) NMOSFET, 1 ms after the SCR firing at 90° (50% modulation). The choke current up to that point rose from 62.6 to 71.3 A, and then suddenly dropped to 40 A in 1 ms, followed by a climb to 140 A in another 1 ms, and finally dropped smoothly to 120 A, and finally suddenly dropped to about 60 A at the next SCR firing at 90°. This corresponds to 122 A RMS. So this could explain the overheating of the choke in the application in question.

I would like to know if this simulation of saturation is reasonably accurate, and if not, how it should be done. It will be necessary to observe the choke current in the application, and obviously, this would be done with an oscilloscope having a channel fully isolated from ground, across a high side shunt, or a non-contact current probe.
Finally, are there some design techniques to build a choke that does not saturate at up to 20 A RMS with 10 A DC, to obtain a reasonably smooth DC output?
(edit) I also ran the simulation replacing the inductor with a 10 mΩ resistor, and that resulted in 32.6 A RMS. So I think there is some resonance happening with 3 mH inductor and 10,000 μF capacitor. Resonant frequency is about 29 Hz.
(edit2) After some thought and analysis I realized that the MOSFET just drew current that was stored in the inductor. So I tried using a voltage controlled switch, but that also did not do what I wanted to simulate saturation. I think perhaps the way to model this would be to add a second coupled coil and throw the switch across it. However, that would probably show the same behavior. What I need to do is measure the total current into the combination of the choke and switch, which is as follows (for 3 mH and 10,000 μF, 50% modulation):
Ideal inductor: I(t) = 19.4  V(out) = 244 V
Saturation 8 ms: I(t) = 20.0  V(out) = 240 V
Saturation 7 ms: I(t) = 21.0  V(out) = 229 V
Saturation 6 ms: I(t) = 30.6  V(out) = 254 V (Resonance?)
(edit3) I found an example of a non-linear inductor model (behavioral flux) and made a simulation of this basic circuit but using a rectangular pulse instead of the phase modulated sine wave. The waveforms show the faster rise of current after saturation (which I set at 20 A). However, I had problems with the simulator (time step too small) when I tried to use ON time of less than 3 ms out of 10 ms period. I set maximum timestep to 1 μs and that worked for this simulation, but it did not for another case. Hopefully this schematic is clear enough.

I am trying to use the Chan transformer model (as suggested by @AConcernedCitizen) but have not yet figured out just how. I have just used LTspice library components and I have little experience with custom models, so I'd like to find a complete example. Here are some resources I'm looking at:
https://groups.io/g/LTspice/topic/creating_a_chan_transformer/73221313?20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,73221313
https://passive-components.eu/modeling-inductors-with-ltspice/
Here is my attempt to use the Chan inductor model, for the MOT transformer choke that measured 3 mH. I used the following values:
Hc=50 (32-72 A/m for electrical steel)
Bs=3 (high value for electrical steel)
Br=1.5 (nominal value for electrical steel)
A=0.001242 (E-I core legs 70 x 18 mm)
Lm=0.000212 (E-I core 70 x 105 x 88 mm)
Lg=0.003 (the gap was 1 mm, but I assume there are three of them
N=75 (this is estimated, but should be 50-100 turns)
Simulation with 300 V pulse 5 ms ON, 10 ms OFF:

I calculated the inductance by using the voltage V across the choke, and the slope of the current A/S, where L = V / (A/S), which in this case is 2.9 mH.

A: Your schematics are hideous and, I'm sorry to say, I won't bother deciphering them. But, since your question is about modelling a saturable choke in LTspice (note the spelling) then you have at least 4 alternatives:

*

*The Chan core uses an inductor with these parameters:
Bs = saturation flux density [T]
Br = remanent flux density [T]
Hc = coercive force [A/m]
A  = core area [m^2]
Lm = core length [m]
Lg = gap length [m]
N  = number of turns

*The Jiles-Atherton core, you can find libraries for it either in the Ltspice Group, or in this user's files collection: Bordodynov. I think there are also examples of usage.

*A behavioural approach made with diodes, e.g. as seen in this IsSpice document.

*The behavioural inductor with Flux=<expr>. E.g. a 1 mH can be modelled as Flux=1m*tanh(x). You will get a smooth tanh()-like curve but, it will not have a hysteresis (zero width).

I'd recommend the Chan core if only for the simplicity of the parameters and its native availability in LTspice.

[edit]
Here are the Chan core and the behavioural Flux inductor:

This is to exemplify how the Flux approach will not give hysteresis, only a fake saturation. Since your latest reply just came, I see you're using some unusual values for Bs. Even Magnesil has 1.6 T (with Br=1.5 Hc=40).

[edit 2]
Your latest comment is a surprise:

But I'm not familiar with your simulator.

It's LTspice, what you're using, but I'll describe the circuit, anyway.
The purpose for this example was to show how the Chan and the Flux core can be used, while showing the differences mentioned at point 4, above. Note that the expression for the Flux differes from the one in your example in that it uses pwrs() instead of a raw power, otherwise there would not be a symmetric response around the zero point (e.g. pwrs() preserves the sign, ** does not).
If you split the schematic vertically, there are two halves: to the left the Chan core and to the right the Flux core. Describing one half: L1 is the Chan core with some (quasi-)random parameters for a (quasi-)random ferrite core. It is driven by a triangular current source, asin(sin(x)), scaled to have the amplitude given by the parameter A (0.1 A here). The voltage across L1 is V(x) which is integrated by G1+C1 with the appropriate time constant to give the voltage V(B1), representing the flux density (T). This is plotted against I(L1), scaled to give the coercive force. Together they are showing the hysteresis graph of the core and you can see that there is, indeed, a hysteresis window.
The Flux core, however, not only it needs adjusting the integration gain (7 in G2 vs 1 in G1), but it also shows no hysteresis window -- it's just a tanh() limiting inductor, no memory.
The two circuits at the bottom are used to determine the values of the inductances, by supplying a unity dI/dt and reading the voltage across as Henry with a 1:1 relation to Volts.


text:
Q: Merge three tables in Select query by rule 3, 2, 1 records from each table Merge three tables in a Select query by rule 3, 2, 1 records from each table as follows:

*

*TableA: ID, FieldA, FieldB, FieldC,....

*TableB: ID, FieldA, FieldB, FieldC,....

*TableC: ID, FieldA, FieldB, FieldC,....

*ID : auto number in each table

*FieldA will be unique in all three tables.

I am looking for a Select query to merge three tables as follows:

*

*TOP three records from TableA sorted by ID

*TOP two records from TableB sorted by ID

*TOP 1 record from TableC sorted by ID

Repeat this until select all records from all three tables.
If some table has fewer records or does not meet the criteria, ignore that and continue with others.
My attempt:
I did it totally through programming way, like cursors and If conditions inside a SQL Server stored procedure.
It makes delay.

A: There is not a lot of information to go with here, but I assume you can use UNION to combine multiple statements.
SELECT * TableA ORDER BY ID DESC OFFSET 3 ROWS
UNION
SELECT * TableB ORDER BY ID DESC OFFSET 2 ROWS
UNION
SELECT * TableC ORDER BY ID DESC OFFSET 1 ROWS

Execute and see if this works.
/AF

A: This requires a formula that takes row numbers from each table and transforms it into a series of integers that skips the desired values.
In the query below, I am adding some CTE for the sake of shortening the formula. The real magic is in the UNION. Also, I am adding an additional field for your control. Feel free to get rid of it.
WITH A_Aux as (
SELECT 'A' As FromTable, ROW_NUMBER() OVER (ORDER BY ID) AS RowNum, TableA.*
FROM TableA
), B_Aux AS (
SELECT 'B' As FromTable, ROW_NUMBER() OVER (ORDER BY ID) AS RowNum, TableB.*
FROM TableB
), C_Aux AS (
SELECT 'C' As FromTable, ROW_NUMBER() OVER (Order BY ID) AS RowNum, TableC.*
FROM TableC
)
SELECT *
FROM (
    SELECT RowNum+3*FLOOR((RowNum-1)/3) As ColumnForOrder, A_Aux.* FROM A_Aux
    UNION ALL
    SELECT 3+RowNum+4*FLOOR((RowNum-1)/2), B_Aux.* FROM B_Aux
    UNION ALL
    SELECT 6*RowNum, C_Aux.* FROM C_Aux
) T
ORDER BY ColumnForOrder

PS: note the pattern Offset + RowNum + (6-N) * Floor((RowNum-1)/N) to group N records together (it of course simplifies a lot for TableC).
PPS: I don't have a SQL server at hand to test it. Let me know if there is a syntax error.

A: You may try this..
    GO

    select * into #temp1 from (select  * from  table1) as t1
    select * into #temp2 from (select  * from  table2) as t2
    select * into #temp3 from (select  * from  table3) as t3

    select * into #final from (select col1, col2, col3 from #temp1 where 1=0) as tb

    declare @i int

    set @i=1
    while( (select COUNT(*) from #temp1)>@i)
    Begin

    ;with ct1 as (
    select ROW_NUMBER() over (order by id) as Slno, * from #temp1
    ),ct2 as (
    select ROW_NUMBER() over (order by id) as Slno, * from #temp2
    ),ct3 as (
    select ROW_NUMBER() over (order by id) as Slno, * from #temp3
    ),cfinal as (
    select top 3 * from #temp1
    union all
    select top 2 * from #temp2
    union  all
    select top 1 * from #temp3
    )
    insert into #final ( col1 , col2, col3 )
    select col1, col2, col3 from cfinal

    delete from #temp1 where id in (select top 3 ID from #temp1)
    delete from #temp2 where id in (select top 2 ID from #temp2)
    delete from #temp3 where id in (select top 1 ID from #temp3)

    set @i = @i+1

    End

    Select * from #final

    Drop table #temp1
    Drop table #temp2
    Drop table #temp3

    GO

First create temp table for all 3 tables with each insert delete the inserted record and this will result you the desired result, if nothing is missing from my side.
Please see to this if this works.

A: From my understanding, I create three temp tables as ta, tb, tc.
select * into #ta from (
select 'A' a
union all
select 'A' a
union all
select 'A' a
union all
select 'A' a
union all
select 'A' a
union all
select 'A' a
union all
select 'A' a
) a

select * into #tb from (
select 'B' b
union all
select 'B'
union all
select 'B'
union all
select 'B'
union all
select 'B'
) b

select * into #tc from (
select 'C' c
union all
select 'C'
union all
select 'C'
union all
select 'C'
union all
select 'C'
) c

If tables match you tables, then the output looks like A,A,A,B,B,C,A,A,A,B,B,C,A,B,C,C,C
T-SQL
declare @TAC int = (select count (*) from #ta)      -- Table A Count = 7
declare @TBC int = (select count (*) from #tb)      -- Table B Count = 5

declare @TAR int = @TAC % 3                         -- Table A Reminder = 1
declare @TBR int = @TBC % 2                         -- Table B Reminder = 1

declare @TAQ int = (@TAC - @TAR) / 3                -- Table A Quotient = (7 - 1) / 3 = 2, is will passed on NTILE
                                                    -- So we gonna split as two group (111), (222)
declare @TBQ int = (@TBC - @TBR) / 2                -- Table B Quotient = (5 - 1) / 2 = 2, is will passed on NTILE
                                                    -- So we gonna split as two group (11), (22)

select * from (
 select *, NTILE (@TAQ) over ( order by a) FirstOrder, 1 SecondOrder from (
  select top (@TAC - @TAR) * from #ta order by a
 ) ta                                               -- 6 rows are obtained out of 7.
 union all
 select *, @TAQ + 1, 1 from (
  select top (@TAR) * from #ta order by a desc
 ) ta                                               -- Remaining one row is obtained. Order by desc is must
                                                    -- Here FirstOrder is next value of previous value.
 union all

 select *, NTILE (@TBQ) over ( order by b), 2  from (
  select top (@TBC - @TBR) * from #tb order by b
 ) tb
 union all
 select *, @TBQ + 1, 2 from (
  select top (@TBR) * from #tb order by b desc
 ) tb

 union all

 select *, ROW_NUMBER () over (order by c), 3 from #tc
) abc order by FirstOrder, SecondOrder

Let me explain the T-SQL:
Before that, FYR: NTILE and Row Number

*

*Get the count.

*Find the Quotient which will pass to NTILE function.

*Order by the NTILE value and static.

Note:

*

*I am using SQL Server 2017.

*If T-SQL works fine, then you need to change the column in order by <yourcolumn>.

redpajama_set_name: RedPajamaStackExchange


text:
Q: SciFi story involving wormhole and insanity I think it was a short story or novella. A wormhole or some other portal is discovered and all attempts at automated study of what is on the other side fail - all recordings come back blank. They send people, but they come back insane so they have people trained as councilors to gently bring them back to reality. Also tied with my memories of this story is the concept of flowers - like maybe in the name or figures in the story.

A: It might be William Gibson's short story Hinterlands. A portal is discovered in the space between Earth and Mars. Those few that are taken through the portal come back insane or dead by their own hand. Sometimes they also bring back invaluable alien tech, so a program is developed to keep sending people and trying to help the ones that come back.

"Ever wonder how they thought this scam up, Toby?" She was squinting
  into the sunlight, east, down the length of our cylindrical country,
  horizonless and green. "They must've had all the heavies in, the
  shrink elite, scattered down a long slab of genuine imitation
  rosewood, standard Pentagon issue. Each one got a clean notepad and a
  brand-new pencil, specially sharpened for the occasion. Everybody was
  there: Freudians, Jungians, Adlerians, Skinner rat men, you name it.
  And every one of those bastards knew in his heart that it was time to
  play his best hand. As a profession, not just as representatives of a
  given faction. There they are, Western psychiatry incarnate. And
  nothing's happening! People are popping back off the Highway dead, or
  else they come back drooling, singing nursery rhymes. The live ones
  last about three days, won't say a goddamned thing, then shoot
  themselves or go catatonic." She took a small flashlight from her belt
  and casually cracked its plastic shell, extracting the parabolic
  reflector. "Kremlin's screaming. CIA's going nuts. And worst of all,
  the multinationals who want to back the show are getting cold feet.
  'Dead spacemen? No data? No deal, friends.' So they're getting
  nervous, all those supershrinks, until some flake, some grinning
  weirdo from Berkeley maybe, he says," and her drawl sank to parody
  stoned mellowness, " 'Like, hey, why don't we just put these people
  into a real nice place with a lotta good dope and somebody they can
  really relate to, hey?' "


text:
Q: How to disable browser back button in django? I am trying to prevent user from going to his logged in page after he logs out.Just tried out what I got from internet and its not working, I am still able to go to back page after I log out.How to prevent this?
from django.views.decorators.cache import cache_control

@cache_control(no_cache=True, must_revalidate=True, no_store=True)
def admin_logout(request):
    logout(request)
    return redirect("user_login")

A: Using @login_required decorator on all the required views will solve the problem

A: If the user has logged out, and the cache is off in the browser, the browser would again hit your view when the back button is pressed. When this happens, you can put an if condition in there to check if session exists or not. If not, then re-route to the desired page.
if not request.session['username'] :
    #redirect to some page


text:
Q: JSF 2 - Bean Validation: validation failed -> empty values are replaced with last valid values from managed bean
I do not understand the behaviour of JSF2 during valdation. Hope someone can help me.
I have a form where the fields are validated after (ajax) submit - ok
If the validation failed a error message is shown - ok
For my example when I enter a valid birthday and the field name is empty an errormessage for name is shown after submit.
Now when I enter a valid name and delete the input from the birthday field an errormessage is show for birthday (that's ok) but now the old 'valid' birthday stands also in the input field!?!
How can I avoid this behaviour?
When I submit an empty field I want to see an errormessage and an empty field...
Here's my sample code:
I use a ManagedBean (TestBean) that contains an EntityBean (Contact). The Contact contains validations per annoations.
public class Contact implements Serializable {
    @NotNull
    @Temporal(TemporalType.DATE)
    private Date birthday;

    @NotNull
    @Size(min=3, max=15)
    private String name;

    //...
}

My ManagedBean:
@ManagedBean
@ViewScoped
public class TestBean implements Serializable {
    private Contact contact;

    @PostConstruct
    void init() {
        System.out.println("init...");
        contact = new Contact();
    }

    public void newContact(ActionEvent ae) {
        System.out.println("newContact...");
        contact = new Contact();
    }

    public void save() {
        System.out.println("save...");
        //TODO do something with contact...
    }

    public Contact getContact() { return contact; }

    public void setContact(Contact contact) {this.contact = contact;}
}

An here my JSF page:
<html xmlns="http://www.w3.org/1999/xhtml"
      xmlns:h="http://java.sun.com/jsf/html"
      xmlns:f="http://java.sun.com/jsf/core" >
 <h:body>
    <h:form>
        <h:panelGrid columns="3">

        <h:outputText value="Birthday: " />
        <h:inputText id="birthday" value="#{testBean.contact.birthday}">
            <f:convertDateTime/>
        </h:inputText>
        <h:message for="birthday" />

        <h:outputText value="Name: " />
        <h:inputText id="name" value="#{testBean.contact.name}"/>
        <h:message for="name" />

        </h:panelGrid>

        <h:commandButton value="submit" action="#{testBean.save}">
            <f:ajax execute="@form" render="@form"/>
        </h:commandButton>

        <h:commandButton value="newContact" actionListener="#{testBean.newContact}"
                         immediate="true">
            <f:ajax render="@form"/>
        </h:commandButton>

    </h:form>
</h:body>
</html>

at last a snippet from web.xml
<context-param>
    <param-name>javax.faces.INTERPRET_EMPTY_STRING_SUBMITTED_VALUES_AS_NULL</param-name>
    <param-value>true</param-value>
</context-param>

<context-param>
    <param-name>javax.faces.VALIDATE_EMPTY_FIELDS</param-name>
    <param-value>true</param-value>
</context-param>

Thanks for some tips

A: Your particular problem is caused by
<context-param>
    <param-name>javax.faces.INTERPRET_EMPTY_STRING_SUBMITTED_VALUES_AS_NULL</param-name>
    <param-value>true</param-value>
</context-param>

and a bug (at least, an oversight) in HtmlBasicRenderer#getCurrentValue() of Mojarra:
if (component instanceof UIInput) {
    Object submittedValue = ((UIInput) component).getSubmittedValue();
    if (submittedValue != null) {
        // value may not be a String...
        return submittedValue.toString();
    }
}

String currentValue = null;
Object currentObj = getValue(component);
if (currentObj != null) {
    currentValue = getFormattedValue(context, component, currentObj);
}
return currentValue;

Normally, the submitted value is set to null when the UIInput component is successfully converted and validated. When JSF is about to redisplay the value, it first checks if the submitted value is not null before proceeding to redisplay the model value. However, with this context parameter, it is null instead of an empty string when it is invalid and thus it will always redisplay the original model value when you remove the initial value of a required field.
To test it, set that context param value to false or remove it altogether. You'll see that it works as intended. However, it will bring back the disadvantage that your model values will be cluttered with empty strings on empty but non-required fields and you'll lose the advantage of using @NotNull annotation of JSR 303 bean validation.
To fix this, you've to alter the first part of HtmlBasicRenderer#getCurrentValue() as follows:
if (component instanceof UIInput && !((UIInput) component).isValid()) {
    Object submittedValue = ((UIInput) component).getSubmittedValue();
    if (submittedValue != null) {
        // value may not be a String...
        return submittedValue.toString();
    } else {
        return null;
    }
}

I've already reported it to Mojarra guys as issue 2266.

A: I'm thinking that during normal use people won't enter a valid date, submit, and then delete the date before submitting again. I realize that you found this during testing, but probably people are just trying to successfully fill out the form and not deleting stuff they already entered, in which case perhaps keeping the last valid value is the best functionality.
If you insist... It seems like the setter method "birthday" is never called because the value is invalid, and then when the page is redisplayed the current value of "birthday" is displayed (the current value being the valid value that was previously saved). Maybe you could write a custom validator that sets the value and THEN validates the value, but this wouldn't really make much sense. You would still have to validate the value first for cases when the users enters a text string like "yesterday" instead of a valid date, and then you would have to set the date to something based on that invalid value, and then you would have to add the message to the FacesContext. So the code would have to do something like the following.

1) validate the date value
2) if it's invalid then set the field to some value which makes sense and add an error message to FacesContext.
3) if it's valid then use it.
That's do-able, but weird because you're changing the value of the field even though the passed in value is invalid...

A: Aaron already descripes the behaviour.
The problem I've been described  exists also by clicking the 'newContact' button. If the first submit is not valid (birthday was entered, name-field is empty) an error message is shown. ok.
Afterwards the 'newContact' Button do not refresh (clear) the view. Although the model was resetted (contact = new Contact()).
I found some tipps here: http://wiki.apache.org/myfaces/ClearInputComponents
Here is my solution:
public void newContact(ActionEvent ae) {
    contact = new Contact();
    contact.setBirthday(new Date()); //for testing only

    resetForm(ae.getComponent());
}

private void resetForm(UIComponent uiComponent) {
    //get form component
    UIComponent parentComponent = uiComponent.getParent();
    if (uiComponent instanceof UIForm)
        resetFields(uiComponent);
    else if (parentComponent != null)
        resetForm(parentComponent);
    else
        resetFields(uiComponent);

}

private void resetFields(UIComponent baseComponent) {
    for (UIComponent c : baseComponent.getChildren()) {
        if (c.getChildCount() > 0)
            resetFields(c);

        if (c instanceof UIInput)
            ((UIInput) c).resetValue();
    }
}

A: Had a similar issue where a value loaded in from the backing bean would get reset when the field was blanked out and another component failed validation. I had to make a slight addition to BalusC's code to make it work.
protected String getCurrentValue(FacesContext context,
                                     UIComponent component) {

        if (component instanceof UIInput && !((UIInput) component).isValid()) {
            Object submittedValue = ((UIInput) component).getSubmittedValue();
            if (submittedValue != null) {
                // value may not be a String...
                return submittedValue.toString();
            } else {
                return null;
            }
        }

        String currentValue = null;
        Object currentObj;

        if ( component instanceof UIInput && ((UIInput)component).isLocalValueSet() )
        {
           currentObj = ((UIInput)component).getLocalValue();
        }
        else {
            currentObj = getValue(component);
        }

        if (currentObj != null) {
            currentValue = getFormattedValue(context, component, currentObj);
        }
        return currentValue;

    }


text:
Q: Linux Shell script to add a user with a password from a list I'm trying to modify a script that read usernames/password from a file which is like this:
user1 pass1
user2 pass2
user3 pass3

I can't get the script to read the space between the users and pass.
What can I use to delimit this space?
This is my code:
for row in `cat $1`
do
  if [ $(id -u) -eq 0 ]; then

      username=${row%:*}
      password=${row#*:}
      #echo $username
      #echo $password

I know I have to change the stuff in ${row%:} and ${row%:}
What do I have to put so it sees the space between user1 pass1 ?

A: It would be easier to split the two fields as you read each line. You can do that with read. It's also better to use a while loop here (a for loop requires to play with $IFS and it would also load the entire file in memory):
#!/bin/bash
if [ "$EUID" -ne 0 ]; then
    echo >&2 "You are not root"
    exit 1
fi

while read -r username password; do
    # do the useradd stuff here
done < "$1"

Notice that I also changed $(id -u) to $UID which should be faster since it does not invoke an external program.


text:
Q: Fluent NHibernate Mapping Components, one indexed, one not indexed Am getting an error on VerifyTheMapping test:

System.ApplicationException: For property 'ANonindexedComponent'
  expected 'Generic.TheDomain.NonindexedComponent' of type
  'Generic.TheDomain.NonindexedComponent' but got '' of type
  'Generic.TheDomain.NonindexedComponent' and
  System.ApplicationException: For property 'IndexedComponent' expected
  'Generic.TheDomain.IndexedComponent' of type
  'Generic.TheDomain.IndexedComponent' but got '' of type
  'Generic.TheDomain.IndexedComponent'

given:
namespace Generic.TheDomain
{
    public class UseAIndexedComponent:DomainEntity<UseAIndexedComponent>
    {
        public virtual string NameTitle { get; private set; }
        public virtual IndexedComponent IndexedComponent { get; private set; }
    }

    public class UseANonindexedComponent:DomainEntity<UseANonindexedComponent>
    {
        public virtual string TitleName { get; private set; }
        public virtual NonindexedComponent
            ANonindexedComponent { get; private set; }
    }

    //we want this one to have a unique index on A B C
    public class IndexedComponent
    {
        public virtual string A { get; private set; }
        public virtual string B { get; private set; }
        public virtual string C { get; private set; }

        public IndexedComponent(){}
    }

    //no unique index on this one...
    public class NonindexedComponent : IndexedComponent
    {
        public NonindexedComponent(){}
    }
}

Fluent NHibernate mappings:
using FluentNHibernate.Mapping;
using Generic.TheDomain;

namespace Persistence.Mappings
{
    public class IndexedMap : ClassMap<UseAIndexedComponent>
    {
        public IndexedMap()
        {
            Table("IndexedComponent");
            Id(x => x.Id).GeneratedBy.GuidComb();
        Map(x => x.NameTitle);
        Component<IndexedComponent>(x => x.IndexedComponent);
    }
}

    public class NonIndexedMap : ClassMap<UseANonindexedComponent>
    {
        public NonIndexedMap()
        {
            Table("NonIndexedComponent");
            Id(x => x.Id).GeneratedBy.GuidComb();
            Map(x => x.TitleName);
            Component<NonindexedComponent>(x => x.ANonindexedComponent);
        }
    }

    public class IndexedComponentMap : ComponentMap<IndexedComponent>
    {
        public IndexedComponentMap()
        {
            Map(x => x.A).CustomSqlType("varchar(1)").UniqueKey("ABC");
            Map(x => x.B).CustomSqlType("varchar(1)").UniqueKey("ABC");
            Map(x => x.C).CustomSqlType("varchar(1)").UniqueKey("ABC");
        }
    }

    public class NonindexedComponentMap : ComponentMap<NonindexedComponent>
    {
        public NonindexedComponentMap()
        {
            Map(x => x.A).CustomSqlType("varchar(1)");
            Map(x => x.B).CustomSqlType("varchar(1)");
            Map(x => x.C).CustomSqlType("varchar(1)");
        }
    }
}

and the test:
[Test]
public void VerifyIndexedMapping()
{
new PersistenceSpecification<UseAIndexedComponent>(Session)
        .CheckProperty(c => c.NameTitle, "jack")
    .CheckProperty(x => x.IndexedComponent, new IndexedComponent())
        .VerifyTheMappings();
}

[Test]
public void VerifyNonIndexedMapping()
{
    new PersistenceSpecification<UseANonindexedComponent>(Session)
        .CheckProperty(c => c.TitleName, "jill")
        .CheckProperty(x => x.ANonindexedComponent, new NonindexedComponent())
        .VerifyTheMappings();
}

A: http://www.nhforge.org/doc/nh/en/index.html#persistent-classes-equalshashcode


text:
Q: Wrong encoded characters when I send utf8 encoded characters from pandas to mysql with latin-1 encoding I'm getting wrong encoded characters when I send utf8 encoded characters from pandas to mysql with latin-1 encoding.
This is a sample value I'm getting in mysql:
Biquíni

This is my code:
df.breakdown_name = df.breakdown_name.str.encode('latin-1')

send_data(DB_ENGINE, MYSQL_USER, MYSQL_PASSWORD, MYSQL_HOST, MYSQL_PORT, MYSQL_DB, MYSQL_TABLE, df)

def send_data(db, db_user, db_password, db_host, db_port, db_name, db_table, df):
    """
        Return data
             1
    """
    # print db
    # print db_user
    # print db_password
    # print db_host
    # print db_port
    # print db_name
    # print query[0:20]
    SQL_ENGINE = '{db}://{db_user}:{db_password}@{db_host}:{db_port}/{db_name}?charset={encoding}'.format(
      db=db,
      db_user=db_user,
      db_password=db_password,
      db_host=db_host,
      db_port=db_port,
      db_name=db_name,
      encoding='latin1',
    )
    engine = create_engine(SQL_ENGINE)
    df.to_sql(name=db_table,con=engine,schema='xxx',if_exists='append', index=False, chunksize=50)

A: I assume that the sample value should be:
print u'Biquíni'.encode('latin_1').decode('utf-8') # Biquíni

Thus, in your code you're doing the inverse transformation:
print u'Biquíni'.encode('utf-8').decode('latin_1') # Biquíni

The problem is that when you implement the chain:

*

*Encode unicode to 'utf-8' bytes.

*Decode those bytes with any 1-byte encoding (like 'latin_1').

for every non-ASCII unicode character at the input you'll always receive wrong characters at the output. That's happen because for such a character there is at least 2-byte value in 'utf-8'.
Let's look at examples:
print ord(u'z') # 122 => ASCII
print repr(u'z'.encode('utf-8')) # 'z', 1 byte
print repr('z'.decode('latin_1')) # u'z'

As we see, for ASCII-character everything works fine, but:
print ord(u'í') # 237 => non-ASCII

import unicodedata

print repr(u'í') # u'\xed'
print unicodedata.name(u'\xed') # LATIN SMALL LETTER I WITH ACUTE

print repr(u'\xed'.encode('utf-8')) # '\xc3\xad' => 2 bytes

print repr('\xc3'.decode('latin_1')) # u'\xc3' - the 1st char
print repr('\xad'.decode('latin_1')) # u'\xad' - the 2nd char

print unicodedata.name(u'\xc3') # LATIN CAPITAL LETTER A WITH TILDE
print unicodedata.name(u'\xad') # SOFT HYPHEN

So, in your code for every non-ASCII character after encoding in 'utf-8' you get 2 bytes which are then decoded with 'latin_1' into 2 characters, and they don't correspond to initial character.
Thereby, the current scheme of your program will generate undesirable results.
I suggest to use the same encoding for encode() and decode() steps in your code.


text:
Q: Application of MLE I was reading about the concept of MLE as to how parameters of a particular assumed distribution are calculated using the MLE approach. But, I wonder what is done next once the value of the parameters with the best possible value is/are calculated. I mean what to do next once the MLE is calculated? How, is it utilized over a new set of points, etc.
For an example say I have a set of data points. Let's denote it by D. By looking at the data and prior experience, I assume Gaussian distribution and try to find out mean and variance. I successfully found those values. Now, the question is how should I use the model data that is available with me?
Any piece of advice would be helpful. If there are post related to this question then, I request you to please provide the link about the same.

A: OK, now we know you're interested in MLEs for normal data. But your goal in
finding these estimates of $\mu$ and $\sigma$ might be. Maybe if I give
a couple of examples, that might help you to refine your question.
Suppose you know that scores on the ABC college entrance exam for this year
are normally distributed with mean $\mu = 350$ and standard deviation $\sigma = 50.$ You have 100 early applications for your college with sample mean $\bar X = 360$ and sample SD $S = 56.$ These sample statistics are based on MLEs.
Estimating average scores. You want to know whether the applicants so
far might be better than the population mean. So you do a one-sample t test
of the  null hypothesis $H_0: \mu = 350$ against $H_a: \mu > 350.$ The
procedure for one-sample t tests is explained in most elementary statistics
texts. Output from Minitab statistical software for such a test on your
data is as follows:
One-Sample T

Test of μ = 350 vs > 350

  N    Mean  StDev  SE Mean  95% Lower Bound     T      P
100  360.00  56.00     5.60           350.70  1.79  0.039

Because the P-value is below 0.05, we say that the sample mean $\bar X = 360$ is significantly larger than the population mean $\mu = 350$ at the 5% level
of significance.
Estimating proportion of qualifying scores. Maybe your college only admits students with ABC scores of 410 or above.
Among the 100 applications at hand, how many have sufficiently high scores?
You may find that there are 15 of them, and estimate that you are on track for 15% of applicants
to qualify. But the fraction of sufficiently high scores among the 100
is not based on MLEs. So this is not the best way to estimate this year's
proportion of qualified students. [We see below that this estimate is to low. Using the number of qualifying scores among the 100 at hand might just as well have given an estimate that is too high.]
Using the MLEs, we ask what proportion of a normal population with mean 360 and SD 56 would have qualifying scores. According to a quick computation in R
statistical software, that turns out to be about 19%. This is a better estimate because the MLEs carry more information about your applications on hand than
does the actual count of qualifying applications.
1-pnorm(409.4, 360, 56)
[1] 0.1888498

Note: it would also be a mistake to look at the proportion of qualifying ABC scores
nationwide because that would be only about 12%.
1 - pnorm(409.5, 350, 50)
[1] 0.1170232

Addendum. A million samples of size $n=100$ were taken from
$\mathsf{Norm}(350, 50).$ The population 88th percentile is 408.75.
For each sample of 100 the 88th percentile was estimated by taking
the 88th sample percentile (sp.88). Also, the 88th population
percentile (pp.88) was found for the distribution estimated using MLEs
$\mathsf{Norm}(\bar X, S).$ The histograms show that the MLE method
is nearly unbiased and has smaller variance than the sample percentile method.

R code for the simulation (but not the histogram) is shown below.
set.seed(920)
mu = 350; sg = 50; n = 100
m = 10^6; sp.88 = pp.88 = numeric(m)
for(i in 1:m) {
  x = rnorm(n,mu,sg)
  sp.88[i] = quantile(x,.88)
  pp.88[i] = qnorm(.88,mean(x),sd(x)) }
mean(sp.88); sd(sp.88)
[1] 407.65
[1] 7.979684  ## larger SD
mean(pp.88); sd(pp.88)
[1] 408.5956  ## closer to 408.75
[1] 6.509124  ## smaller SD


text:
Q: Is it okay to have duplicate IDs in multiple contact forms if needed? Can anyone help with this? I really need to get it sorted. Please read fully and look at the site to see what i mean specifically...
I have a simple contact form for each item (10 items+) on a, for all intents and purposes, an item gallery.
But when I run through a validator it says i have duplicate IDs. The forms all work as intended but I'm guessing this will become a problem at some point
I have a single "html_form_send.php" which all my forms point to is there a way to get round the duplicate IDs?
Can i class the ID's instead? If so do i need to amend the "html_form_send.php" and "....css" files to accommodate this? If so in what way?
If you need to page to look at its http://www.flowright.net/fullrange
<form name="htmlform" method="post" action="html_form_send.php">

                Full name *

                <input  type="text" id="first_name" maxlength="50" size="30">

                Company *

                <input  type="text" id="last_name" maxlength="50" size="30">

                Email Address *

                <input  type="text" id="email" maxlength="80" size="30">

                Telephone Number *

                <input  type="text" id="telephone" maxlength="30" size="30">

                Message / Query *

                <textarea id="comments" maxlength="1000" cols="40" rows="6">**REF SITRANS MAG 1100**</textarea>

                <input type="submit" value="Submit">

</form>

A: First of all, I don't see duplicate id's in the code you posted, but you did mention you ran it through some sort of validator. You should really specify what validator you are using and what it's response is. Also, your label tag says "Full Name" but your id is for first_name, and your label for "Company" has an id for last_name. You should also be declaring the correct types for your inputs (type="email", type="tel")
I am assuming you are using one of the id's in your code somewhere else in the remainder of your body, but either way, it is okay to use class instead of id. It really comes down to what you are trying to do. The purpose of id and class is mostly for css specificity. When using it with javascript, using css selectors is just one of many ways to select DOM elements (but it is probably the easiest).


text:
Q: Using "Swift into ObjectiveC" but Generated Interface Header is not found I am struggling with the attempts to add MyClass.swift into ViewController.m
I have looked at this solution but without any success.
The code build succeeds but the Project-Swift.h is marked as "not found", this is quite annoying and I can not find the source of the issue.
Project Build setup:
1.Project uses $(PROJECT_NAME)-Swift.h as Generated Interface Header
2.Target_One uses Project-Swift.h as Generated Interface Header
3.Target_Two uses Project-Swift.h as Generated Interface Header
Switching Defines Module to YES/NO in Project&Targets in BuildSettings makes no difference, so does Always Embed Swift Libraries
I check in /DerivedData and can see that the file is there, still xCode is really confused of this file when it builds OK but the file is marked as "not found"


text:
Q: Is this a good server/subdomain set up At my work I am in charge of a few servers that are VPS's. What I generally do is give each server a name (at the moment I'm using the names of the 5 wizards from Tolkien's works).
Then in the zone file for our organization's domain, I set up a records for those servers. For example:
alatar.ourorganization.com 12.34.56.78
gandalf.ourorganization.com 98.76.54.32

And so on. Then each of these servers has its hostname set to alatar.ourorganisation.com or whatever.
Sometimes we want to have various things on those servers, like a survey or a reporting system or something like that, so to set that up I usually use a cname to set up the new subdomain, so for example:
reports.ourorganization.com gandalf.ourorganization.com

Is this a reasonable way of setting this kind of thing up? Are there any problems that could arise from doing things this way?

A: its fine. that's what we do also.
Just bear in mind if you want a server to be a mail server the mx record needs its real name, not a cname.


text:
Q: What is the proper way to utilise sessions for authentication in php? Currently what I do is store the user's username after a sucessful login into a session variable.
$_SESSION['session_loggedin'] = $post_username;

post_username is the POST from the submit form.
Then i use this session variable to check if it is set, to see if a user is logged in. I use the value of this variable to show user-specific content.
                <?php
                if (isset($_SESSION['session_loggedin'])) {
                    ?>
                        LOGOUT
                     <?php
                }
            ?>

Is this how sessions are meant to be used? Is this a right way to securely show content? How do I prevent hijacking?
Thanks

A: You're almost there! But you are supposed to store the user's id on session. You query the database where username is equal to post username and password is equal to post password, the count the number of rows return! From there you store the user's id on session.

A: To see how PHP sessions are used please go to:http://www.w3schools.com/php/php_sessions.asp
. No since i could go onto your other pages (assuming you have some) and insert a username i.e. if John Smith Logged in with a Password:password thats fine but i could go onto you website and put John Smith as $_SESSION['session_loggedin'] and be away with his account. You may ask but how will he know about $_SESSION['session_loggedin'] if he is smart he would create a user and capture the sessions and see what he needs to input to gain control of another account. Instead of a username i would recommend a hashcode!
See this for hijacking:What is the best way to prevent session hijacking?


text:
Q: catchment area using SAGA and Arcgis 10.2 I would like to calculate catchment area. I used D8 algorithm and running in both arcgis 10.2 and SAGA GIS 2.1.2. However, the results from two software are different. The lowest value of catchment area in arcgis = 0; high value = 109325, while SAGA GIS lowest value= 625, highest value=47820. I think the result should be similar and maybe I do something wrong. Could you please explain what I did wrong? These are what i am doing,
the input: DEM25
In arcgis, to calculate catchment area: Dem25, then run tool flow direction, then run tool flow accumulation, then 25 (cell size) * 25 * flow accumulation
In SAGA GIS, terrain analysis -- hydrology --catchment area (parallel), elevation = dem25, sink routes=no set, weight = not set, catchment area = create, Step =1, linear flow = no tick, linear flow threshold =500. convergence = 1.100000001
Thanks so much for your help.

A: ArcMap uses the D8 routing algorithm for determining flow directions between cells, with no convergence exponent since it is a single direction algorithm, while the default in SAGA GIS (I believe - at least it is in mine) is MFD with the convergence you specify.
These methods differ and therefore will generate different catchment area results.
You could run SAGA GIS with the D8 algorithm (and a convergence = 1), and this should make the results match more closely, but I have the feeling that even then they will not be identical.  Different implementations of the same algorithm (differences in the way they are coded up) should still lead to differences in the results.
Hope this helps.

Edit:  See the comment from @WhiteBoxDev below as well.


text:
Q: How to retrieve all variants of a lexeme in Java? I am searching for a way to retrieve all variants of the lexeme of a specific word.
Example: running -> (run, runs, ran, running…)
I tried out Stanford NLP according to this post. However, the lemma-annotator only retrieves the lemma (running -> run), not the complete set of variants. Is there a way to do this with Stanford NLP or another Java Lib/Framework?
Clarification: I do not search for a stemmer. Also, I would like to avoid programming a new algorithm from scratch to crawl WordNet or similar dictionaries.

A: The short answer is that a standard NLP library or toolkit is unlikely to solve this problem. Like Stanford NLP, most libraries will only provide a mapping from word --> lemma. Note that this is a many-to-one function, i.e., the inverse function is not well-defined in a word space. It is, however, a well defined function from the space of words to the space of sets of words (i.e., it's a one-to-many mapping in word-space).
Without some form of explicit mapping being maintained, it is impossible to generate all the variants from a given lemma. This is a theoretical impossibility because lemmatization is a lossy, one-way function.
You can, however, generate a mapping of lemma --> set-of-words without much coding (and definitely without coding a new algorithm):
// Java
Map<String, Set<String>> inverseLemmaMap = new HashMap<>();

// Guava
Multimap<String, String> inverseLemmaMap = HashMultimap.create();

Then, as you annotate your corpus using Stanford NLP, you can obtain the lemma and its corresponding token, and populate the above map (or multimap). This way, after a single pass over your dataset, you will have the required inverse lemmatization.
Note that this will be restricted to the corpus/dataset you are using, and not all words in the English language will be included.
Another note is that people often think that an inflection is uniquely determined by the part of speech. This is incorrect:
String s = "My running was beginning to hurt me. I was running all day."

The first instance of running is tagged NN, while the second instance is the present continuous tense of the verb, tagged VBG. This is what I meant by "lossy, one-way function" earlier in my answer.


text:
Q: Using UpdateResource() to update one's own resources I have an application whose resources need to be updated externally from time to time. I want the application to update it's own resources.
The problem with UpdateResource() is that it cannot be used on a currently executing process. Any ideas how I can update resources from the application itself instead of writing an external app to handle this?
regards,
andy

A: You should move the resources to an external DLL and unload it while updating and reload
it back.

A: This seems to be not a good solution to update the application resources. Programs are installed with Administrator rights. A standard user has no write access to the program executable files. Consider saving the data you need in a user directory. E.g. %LOCALAPPDATA%. If you really need resources you can also put a resource-only DLL there. Then it is easy to update even in a multi-user scenario.

A: You can't modify your own executable.
Also, you must not do this. What if you're running from read-only drive?
For storing program state, there is %APPDATA%, %LOCALAPPDATA% and registry.


text:
Q: How do I disable search-as-you-type text in Ubuntu 17.10 and later? I cannot find enable-interactive-search in
gsettings set org.gnome.nautilus.preferences enable-interactive-search true.

I searched using
gsettings list-recursively | grep search

and
gsettings list-recursively | grep interactive

but no such option.
Can you help me disable this in Ubuntu 17.10 and later?

A: If you really want to have this functionality in Nautilus 3.26 (which is right now the current version in Ubuntu 17.10), check out my step-by-step guide to get it in this question: Traditional search-as-you-type on newer Nautilus versions to build your own version of Nautilus with the support patched back in!

There is actually a way to get this in Nautilus 3.26 (the current
  version in Ubuntu 17.10), which is building your own version from
  source using a patch provided by the awesome Arch Linux community.
  Luckily the great build system in Ubuntu makes this quite easy.
  Here are the steps. I'll assume you'll be working in
  ~/bld-nautilus-typeahead. Open the terminal and do the following:
# install some necessary tools
sudo apt-get install git

# Create your work directory and go there
mkdir bld-nautilus-typeahead ; cd bld-nautilus-typeahead

# Clone the repository holding the needed patch:
git clone https://aur.archlinux.org/nautilus-typeahead.git

# Make sure the source repositories for the main archives are available:
sudo nano /etc/apt/sources.list

# If the deb-src line for the main repository (usually around line 6)
# is commented out (starts with a #)
# uncomment it (remove the #) and save the file.
sudo apt-get update

# Install the build dependencies
sudo apt-get build-dep nautilus

# Retrieve the sources for Nautilus
apt-get source nautilus

# Source should now be in the 'nautilus-3.26.0/' folder. Go there
cd nautilus-3.26.0/

# and Apply the patch from arch-linux
patch -p0 < ../nautilus-typeahead/nautilus-restore-typeahead.patch

# Build the package from source
dpkg-buildpackage -rfakeroot -uc -b

# This will take a little time. If everything goes well, the related packages will end up in the parent directory. Go there
cd ..

# and install the required packages
sudo dpkg -i nautilus_3.26.0-0ubuntu1_amd64.deb nautilus-data_3.26.0-0ubuntu1_all.deb

Start a new Nautilus. Type some letters. Experience that feeling of
  joy that happens when the file you intended gets selected.

A: Gnome decided to remove the navigation feature type-ahead and replace it with search. They fail to see this is a big problem for many users.
Up until Ubuntu 17.10 Canonical provided a patched version of Nautilus that retained this feature. This is not the case anymore and type-ahead is gone for good.
You are of course welcome to join the Nautilus mailing list and state your reasons why this feature is still needed. I don't see much use in that, though. One of the maintainers told me "This has been discussed to death and we have likely heard all the arguments."

A: Woohoooo! There is a brand new modified version of Nautilus.
See https://launchpad.net/~lubomir-brindza/+archive/ubuntu/nautilus-typeahead
I've been waiting for a long time before I find this feature again. Hope that Nautilus developers will consider adding it as an option in the app.

A: I've participated to long debates with the Nautilus developers and we were not able to be convincing enough to have this feature as it was before. The only response was that it was going to be improved in the future.
I like Nautilus but this change made it unusable for me. That drives me crazy.
My solution is simple: switch to another software. I now use Nemo which is fork of Nautilus.


text:
Q: Twitter Digits Web CDN SDK redirecting to insecure CDN URL We have integrated Twitter Digits on our site using their following CDN SDK link
https://cdn.digits.com/1/sdk.js
Sometimes internally it is redirected to following link.
https://cdn-o.digits.com/1/sdk.js
Since this URL is ENSECURE I get following error, so the Digits is not defined.
GET https://cdn-o.digits.com/1/sdk.js net::ERR_INSECURE_RESPONSE
Seems the issue with digits CDN.
or is there any way I can resolve this issue?

A: This was happening because of WordPress HTTPS plugin.
We uninstalled it and everything started working normal.


text:
Q: AKS Cluster with virtual node enabled and without virtual node enabled I wanted to install Kubeflow into the Azure, So I started off creating an Azure Kubernetes Cluster(AKS) with a single node(B4MS virtual machine). During the installation, I didn't enable the virtual node pool option. After creating the AKS cluster, I ran the command "$ kubectl describe node aks-agentpool-3376354-00000" to check the specs. The allocatable number of Pods were 110 and I was able to install Kubeflow without any issues. However, sometime later I wanted an AKS Cluster with virtual node pool enabled so I could use GPUs for training. So I deleted the old Cluster and created a new AKS Cluster with the same B4MS virtual machine and with the virtual node pool option enabled. This time when I ran the same command as above to describe the node specs, the allocatable number of Pods were 30 and the kubeflow installation failed due to lack of pods to provision.
Can someone explain me why the number of allocatable Pods change when the virtual node option is enabled or disabled? How do I maintain the number of allocatable Pods as 110 while having the virtual node pool option enabled?
Thank you in advance!

A: Virtual Node Pool requires the usage of the Advance Networking configuration of AKS which brings in AZURE CNI network plugin.
The Default POD count per node on AKS when using AZURE CNI is 30 pods.
https://learn.microsoft.com/en-us/azure/aks/configure-azure-cni#maximum-pods-per-node
This is the main reason why you are now getting 30 MAX pods per node.
This can be updated to a bigger number when using the AZ CLI to provision your cluster.
https://learn.microsoft.com/en-us/cli/azure/ext/aks-preview/aks?view=azure-cli-latest#ext-aks-preview-az-aks-create
--max-pods -m
The maximum number of pods deployable to a node.


text:
Q: Redis strategy for activity log data Our application requires an activity log of actions that a user has done. For example:
3/19/2013
-------------
9:37 AM Chuck Norris scrapped 3 robots.
9:15 AM Chuck Norris fought 3 robots.
Currently this is done with a gigantic query that compiles all of a user's activity on a given day. The query is so slow that that application literally forbids you from looking at more than a single day's activity at a time. These results could be aggressively cached in Redis.
My question is: what is a good Redis key strategy for storing these activities?
I need to be able do a lookup primarily based on the user ID and a date range. I've played with the following possibilities, but nothing has really leaped out at me as the optimal solution.
-- psuedo-code, obviously --
HMSET user:5345:activity date "3/19/2013" time "9:37 AM" action "Some action"
LPUSH user:5345:activity (json)
LPUSH user:5345:activity:2013:03:19 (json)

All three of these have significant tradeoffs. Assume a user hasn't logged in for four months - they might actually want to see their activity from four months ago at the top of the list. The second option gives me that, but there are trade-offs in that direction as well (like not being able to specifically query by date).
Any thoughts on a better way to approach this?

A: Interesting challenge (!) because dates are sparse.
Here's a thought: if you can find a way to represent dates as numbers (minutes past the year 2000?), then you could try a Sorted Set. The set would be keyed on the user ID. The score would be the numeric date representation. The value would be the log entry.
Then you can use ZRANGEBYSCORE to get log entries within a date range. Also, it should be easy/efficient to "get the N most recent log entries".
Example: ZADD user:23432:activity 1363800946 data

A: I'm one of the authors of the Feedly package which was mentioned.
We built a solution for the 2nd largest online fashion community and open sourced our approach:
https://github.com/tschellenbach/Feedly
It's currently the largest open source library aimed at solving this problem.
You can use both Redis and Cassandra as storage backends.
Our redis storage backend uses sorted sets for data storage.
https://github.com/tschellenbach/Feedly/tree/master/feedly/storage/redis
In addition we use pubsub capabilities for the realtime components.
The same team which built Feedly also offers a hosted API, which handles the complexity for you. Have a look at getstream.io There are client libraries for Python, PHP, Node and Ruby. (This is based on Cassandra and therefore substantially cheaper to operate than running your own system using Redis)
In addition have a look at this high scalability post were we explain some of the design decisions involved:
http://highscalability.com/blog/2013/10/28/design-decisions-for-scaling-your-high-traffic-feeds.html
This tutorial will help you setup a system like Pinterest's feed using Redis. It's quite easy to get started with.
To learn more about feed design I highly recommend reading some of the articles which we based Feedly on:

*

*Yahoo Research Paper

*Twitter 2013 Redis based, with fallback

*Cassandra at Instagram

*Etsy feed scaling

*Facebook history

*Django project, with good naming conventions. (But database only)

*http://activitystrea.ms/specs/atom/1.0/ (actor, verb, object, target)

*Quora post on best practises

*Quora scaling a social network feed

*Redis ruby example

*FriendFeed approach

*Thoonk setup

*Twitter's Approach


text:
Q: Xpath sibling condition (or) I'm trying to find the first preceding-sibling of a node with OR condition.
For example , i have a node A and i try to get the first preceding-sibling B but if there is no B then get first preceding-siblingSPAN.
I don't know how to do the OR condition:
This is what i tried :
A//preceding-sibling::B|SPAN[1]
A//preceding-sibling::B[1]|SPAN[1]
A//preceding-sibling::B[1]|preceding-sibling::SPAN[1]
A//(preceding-sibling::B[1]|preceding-sibling::SPAN[1]) // Xpath invalid
A//preceding-sibling::(B|SPAN)[1] // Xpath invalid
.... and many more

In short I'm looking for :  Node -> preceding-sibling::NodeType1[1] OR preceding-sibling::NodeType2[1]
Is there a way to achieve this with Xpath or i must try one and if there is no result try the other ?
EDIT:
The good answer to this question is I think StuartLC answer (I didn't try it).
If you are looking for the first node between B and SPAN then used har07 answer.

A: As per Michael's comment, note that *[name()='B' or name()='SPAN'] will return the first preceding-sibling which is either B or SPAN.
If I understand your question correctly, you want to 'fallback' to SPAN only IFF there are no B preceding-siblings of A at all. AFAIK you will need to repeat the query with a union:
(A/preceding-sibling::B[1] | A/preceding-sibling::SPAN[1])[1]

A: You can try this way instead :
A//preceding-sibling::*[name()='B' or name()='SPAN'][1]


text:
Q: Drag and drop inside an image with kineticjs I want to drop the star on a tree and detect which tree the star has been dropped on so I can highlight and select it eventually. But I couldn't get more than just moving the star.
I chose for kineticjs because it works with touchscreen so I don't want to use something else if possible. (unless it works with both touchscreen and mouse too)
This is my javascript:
<script defer="defer">
  function drawImage(imageObj) {
    var stage = new Kinetic.Stage({
      container: "star",
      width: 900,
      height: 500
    });
    var layer = new Kinetic.Layer();

    // star
    var star = new Kinetic.Image({
      image: imageObj,
      x: 376,
      y: 30,
      width: 40,
      height: 46,
      draggable: true,
      draw: false
    });

    // add cursor styling
    star.on('mouseover', function() {
      document.body.style.cursor = 'pointer';
    });
    star.on('mouseout', function() {
      document.body.style.cursor = 'default';
    });

    layer.add(star);
    stage.add(layer);
  }
  var imageObj = new Image();
  imageObj.onload = function() {
    drawImage(this);
  };
  imageObj.src = 'http://upload.wikimedia.org/wikipedia/commons/d/df/Star_icon_1.png';

</script>

Please see the fiddle for the full code:
http://jsfiddle.net/hpq7rpnh/1/

A: Add the trees as their own objects, and then you can check collision between stars and trees:
var starLayer = new Kinetic.Layer(); // its own layer, index should be above tree layer
var treeLayer = new Kinetic.Layer(); // its own layer

stage.add(treeLayer);
stage.add(starLayer);

var tree = new Kinetic.Rectangle( ... );

treeLayer.add(tree);

var tree2 = new Kinetic.Rectangle( ... ); // another tree at another coordinate
treeLayer.add(tree2); // assuming you have a layer for Trees ONLY already

later on, when the user drops the star, you need to check collision
// sample bounding box collision detection
function checkCollision(){
    var trees = treeLayer.getChildren(); // gets all the trees
    for(tree in trees)
        if(star.xCoord > tree.xCoord && star.xCoord + star.width < tree.xCoord + tree.width ... same for y coordinates)
}


text:
Q: How to programmatically determine the currently active App in Android Is there a way to find the app whose activity is currently active in android? Does android send a broadcast when a new app/activity is active?

A: Call ActivityManager.getRunningAppProcesses(), then iterate through the ActivityManager.RunningAppProcessInfo objects and look for the one(s) with IMPORTANCE_FOREGROUND.
I'm not sure if there are situations (tablet? Laptop dock?) when you can have multiple foreground activities.


text:
Q: LINQ for CRM how to use C# list in where clause I need to use a C# list in LINQ where clause. I can do query with one value in where clause, please guide how to use a C# list in LINQ query.
Below is my code:
var usersList= new List<string>();

usersList.Add("User1");
usersList.Add("User2");

(from u in UserSet
where u.FullName.Equals("any user from usersList")
select u.FullName).Take(3).Dump();

Basically how to tell LINQ to check every value in usersList
Edit:
I m doing this LINQ for CRM
Thanks

A: You can use Contains
var usersList= new List<string>();
usersList.Add("User1");
usersList.Add("User2");

...

(from u in UserSet
where usersList.Contains(u.FullName)
select u.FullName).Take(3)

Enumerable.Contains Method (IEnumerable, TSource)

Determines whether a sequence contains a specified element by using
  the default equality comparer.

Update
I cant see a way to use Contains with a in memory list. However if the user list is not too big you could do it after the fact in memory
var results  = (from u in UserSet
               select u.FullName);

var results (from r in results
             where usersList.Contains(r)
             select r).Take(3);

There has to be a better way though.

A: You cannot use Contains method with LINQ for Dynamics CRM.
The General is correct, you can bring the entire entity set into memory and then perform your contains method. However, this is inefficient - especially when the entity sets increase after the system has gone live.
I have made the assumption that you are using early bound context, that is why you want to use LINQ, if this is correct then an option would be to do something like the following:
var usersList= new [] {
 "User1",
 "User2"
};

var systemUserQuery = new QueryExpression{
    EntityName = SystemUser.EntityLogicalName,
    ColumnSet = new ColumnSet("fullname","domainname" ....), //explicitly retrieve attributes
    Criteria = {
        Conditions = {
            new ConditionExpression("fullname", ConditionOperator.In, userList)
        }
    }
};

var usersResponse = organizationService.RetrieveMultiple(systemUserQuery);
var systemUsers = usersResponse.Entities.Select(s => (SystemUser)s).ToArray();

This does not use LINQ in the expression to retrieve the data, however the query is executed remotely, and you do not need to retrieve the entire entity set. In the statement var systemUsers = usersResponse.Entities.Select(s => (SystemUser)s).ToArray(); The object type Microsoft.Xrm.Sdk.Entity will be cast as the Early Bound Type SystemUser and you can continue to use the objects as you any other Early Bound object.
When using the ConditionOperator.In operator, the value supplied will need to be an array. You can declare the userList variable as List<string> as in your example. However, you will need to cast it before adding the object into the query.
Just a note the line ColumnSet = new ColumnSet("fullname","domainname" ....) is where you select the attributes to retrieve in the request. If you do not specify the attribute here the property will be the default value after the cast.


text:
Q: Store values from different variables in one variable I am using a dataset that stores country-specific information in different variables, I want to combine this into a single variable. It feels like this should be such an easy exercise, but I cannot figure it out and I can't find any answers here.
Say the data looks like this:
df <- data.frame(country = c("BE", "BE", "BE", "NL", "NL", "NL"),
                 year = c(2010, 2010, 2010, 2010, 2010, 2010),
                 party_NL = c(NA, NA, NA, "A", "B", "B"),
                 party_BE = c("C", "D", "E", NA, NA, NA))

  country year party_NL party_BE
1      BE 2010     <NA>        C
2      BE 2010     <NA>        D
3      BE 2010     <NA>        E
4      NL 2010        A     <NA>
5      NL 2010        B     <NA>
6      NL 2010        B     <NA>

What I need is the following:
  country year party_NL party_BE party
1      BE 2010     <NA>        C     C
2      BE 2010     <NA>        D     D
3      BE 2010     <NA>        E     E
4      NL 2010        A     <NA>     A
5      NL 2010        B     <NA>     B
6      NL 2010        B     <NA>     B

Guessing some loop would have to be applied. Once again, it sounds so easy that I apologize in advance.
Thanks

A: You can use coalesce in dplyr:
df %>%
  mutate(party = coalesce(party_NL, party_BE))

Output:
  country year party_NL party_BE party
1      BE 2010     <NA>        C     C
2      BE 2010     <NA>        D     D
3      BE 2010     <NA>        E     E
4      NL 2010        A     <NA>     A
5      NL 2010        B     <NA>     B
6      NL 2010        B     <NA>     B

A: Use mutate from dplyr, this will do the trick:
require(dplyr)
df %>%
  mutate(party = ifelse(is.na(party_NL), party_BE, party_NL))


text:
Q: Why does LASSO ignore a predictor that has predicting power and NOT correlated with other predictors? I have a linear regression problem for my car fleet data, where $y$ is the change in rental price and $X$ is a design matrix with around 30 columns (predictors).
Most of the predictors are continuous real values, while only of them, $x_{22}$ is an 'encoded' categorical feature, which has only 2 values, -1 and 1.
when I do a simple $lm(y  =  0 + x_{22})$, I see a R-squre similar to other predictors. However, when I use LASSO to fit it, this feature is dropped -- unless I make the penalty term very small
I cannot make sense of this, because the correlation between this feature and other features is very low. In other words, this feature is very unique. It will be very nice to have it included in the final model.
Can someone explain to me what's the possible rationality for LASSO to drop it? Does this reveal some limitation/disadvantage of LASSO? or based on the result ($x_{22}$ is dropped) there is something I am not awared of yet about my features. One of my hypothesis is something like, although no exact correlation, there is some other form of relationshipt between $x_{22}$ and other predictors.

A: To have a valid analysis of change in rental price would require demonstration that the change is independent of the initial price.
To your lasso question, bootstrap the lasso analysis and examine the volatility of the features selected.  In other words, first see if lasso is self-consistent before seeing if lasso is consistent with another method.


text:
Q: Get Linux file descriptor of FileChannel Is there a way to get the Linux file descriptor for an opened FileChannel?
I need it to call mount.fuse -o fd=... (for implementing FUSE).
As a hacky workaround, I'm doing:
var pid = ProcessHandle.current().pid();
var fd = Files.list(Path.of("/proc/"+pid+"/fd")).count();
var fc = FileChannel.open(path);
System.out.println("file descriptor: " + fd);

Note that two file descriptors appear. One for path and another socket. I'm using the first one. What is the socket for?

A: You can use reflection to get the file descriptor from a RandomAccessFile:
long pid = ProcessHandle.current().pid();
long guessedFd = Files.list(Path.of("/proc/"+pid+"/fd")).count();

var file = new java.io.RandomAccessFile(FUSE_DEVICE_PATH.toFile(), "rw");
FileChannel fc = file.getChannel(); // Use FileChannel for fast NIO
var javaFd = file.getFD();
try {
    Field f = FileDescriptor.class.getDeclaredField("fd");
    f.setAccessible(true);
    var trueFd = (int) f.get(javaFd);
    return trueFd;
}
catch (InaccessibleObjectException | NoSuchFieldException | IllegalAccessException e) {
    return guessedFd;
}

On Java 11+ you will the jvm option --add-opens=java.base/java.io=ALL-UNNAMED at startup. Tested on Java 18.

In Java 17 you can use SharedSecrets instead of reflection:
FileDescriptor javaFd = ...
try {
    int trueFd = jdk.internal.access.SharedSecrets.getJavaIOFileDescriptorAccess().get(javaFd);
    return trueFd;
}
catch (IllegalAccessError e) {
    return guessedFd;
}

You will need to add --add-exports=java.base/jdk.internal.access=ALL-UNNAMED when compiling and running. See for Maven.


text:
Q: WSO2 IOTS: How to use Event Store to store device history I'm trying to find out how WSO2 CEP work, Event Stream is simple, but what is Event Store? It can store device history but in where? I'm thinking about Spark and it only store data in memory, right? If I want to store device history in RDMS, what I can do? Thanks

A: If I explain briefly how WSO2 CEP works or rather how CEP steaming engine works(which powers ITOS), There are events streams which carry device data, etc. You can write queries to consume these streams, do some manipulations/operations and produce another event stream as output. This will be done in real-time. Siddhi is the language which you will use to write these queries. Siddhi provides feature called event tables[1] where you can store events in either in-memory or RDBMS tables and later use it.
Event Store is a place where you can directly persist[2] incoming events to a RDBMS. But this data is mainly for the purpose of batch analytics which is powered by Apache Spark. With Event Store you can store the full stream and then later run hourly/daily Spark queries on them to extract stats. And of course you can do the same for data you stored in RDBMS though Siddhi which is mentioned earlier if needed.Following references will help to understand more.
[1] https://docs.wso2.com/display/DAS300/Siddhi+Query+Language#SiddhiQueryLanguage-EventTable
[2] https://docs.wso2.com/display/DAS300/Persisting+Data+for+Batch+Analytics
https://docs.wso2.com/display/IoTS100/How+WSO2+IoT+Server+Function
https://docs.wso2.com/display/DAS300/Architecture


text:
Q: No seller_id in dimensions but can create a count distinct measure on it in kylin sample cube CDH: 5.7  Kylin: 1.5.2_CDH_5.7
In kylin sample cube kylin_sales_cube, there is a measure named SELLER_CNT_HLL which is COUNT_DISTINCT and uses the column seller_ID
but when I edit it I can't select the seller_id because there is no column named seller_id in the dropdown list and the seller_id is not a dimension in the cube
I am little confused about this, I don't know whether this is a mistake but this is what I want, the cardinality of seller_id is huge at actual situation I don't want seller_id be a dimension but I want the COUNT DISTINCT of seller_id and filter by a single seller_id
Please help me out with the confusion,thanks

A: From kylin's dev mailing list I got the answer
The "seller_id" was missed in data model's "measure" columns; Please edit
the data model and add it there, and then when edit the cube, you will be
able to select it the measure.
http://apache-kylin.74782.x6.nabble.com/No-seller-id-in-dimensions-but-can-create-a-count-distinct-measure-on-it-in-kylin-sample-cube-td4813.html


text:
Q: pheatmap column cluster annotation I have following code for the heat map generation using "pheatmap" package. I am successful creating heat map but having very hard time to annotate the column. I have 9 cell types and each cell type has 3 replicates.
Here is the data. I have labeled samples as 01_01,01_02,01_03 to 09_01, 09_02,09_03.
I would like to have column annotation as "A", "B", "C", "D","E", "F", "G", "H", "I". pheatmap with clustering row and column
I tried my best to annotate cluster column but wasn't successful. I really appreciate if someone can explain how to do customize column annotation.
Data:
my data file
library(readxl)
library(pheatmap)
library(tibble)
library(dplyr)

read file
test<-read_excel("~test.xlsx",1)

test%>%
   column_to_rownames("Metabolite") %>%
   as.data.frame()

test<-as.data.frame(test)
rownames(test)<-test$Metabolite
test$Metabolite<-NULL

    pheatmap(test,cluster_rows=TRUE,cluster_cols=TRUE,breaks=NA,scale="none",legend=TRUE,color=colorRampPalette(c("navy","white","firebrick3"))(50),margins=c(5,10),fontsize_row=4,fontsize_col=4,cellwidth=5,cellheight=5)


text:
Q: Does Amazon Redshift support extension dblink? CREATE EXTENSION dblink;

I tried to create an extension dblink in AWS, but I receive this error.

ERROR: syntax error at or near "EXTENSION" LINE 1: create extension
  dblink;

A: The answer is "Yes", but you have to use Amazon RDS as a 'cache' by leveraging the Dblink feature of RDS. This allows us to make any Amazon Redshift table available as a relation in RDS, without the need to explicitly copy that data over. Dblink handles moving the data at the block level.
We've used the approach described in the link that John has provided above. We did a detailed write up of how we did it in a blog post "Have Your Postgres Cake with Amazon Redshift and eat it, too."
https://www.intermix.io/blog/using-amazon-redshift-with-amazon-rds/

A: While AWS Redshift is based on Postgresql, it has various enhancements/alterations to integrate both AWS infrastructure and also a very different backend from standard Postgresql. In order to create a DB Link, you don't need to (in fact, you can't) load the DB Link extension.
Instead, AWS have replaced the standard Postgresql CREATE SERVER AND IMPORT FOREIGN SCHEMA statement with their unique CREATE EXTERNAL SCHEMA - this creates your DB links for you:
CREATE EXTERNAL SCHEMA [IF NOT EXISTS] local_schema_name
FROM { [ DATA CATALOG ] | HIVE METASTORE | POSTGRES | MYSQL | KINESIS }
[ DATABASE 'database_name' ]
[ REGION 'aws-region' ]
[ URI 'hive_metastore_uri' [ PORT port_number ] ]
IAM_ROLE { default | 'arn:aws:iam::<AWS account-id>:role/<role-name>' }
[ SECRET_ARN 'ssm-secret-arn' ]
[ CATALOG_ROLE 'catalog-role-arn-string' ]
[ CREATE EXTERNAL DATABASE IF NOT EXISTS ]

This does everything in one statement. Full documentation can be found here

A: Yes and No.
Amazon Redshift itself does not provide dblink. However, you can install dblink on a different PostgreSQL server and use it to connect to Amazon Redshift.
See: JOIN Amazon Redshift AND Amazon RDS PostgreSQL WITH dblink


text:
Q: Crontab: Is it possible to activate `pyenv` as soon as reboot? Environment :  Linux Ubuntu 16.04, pyenv, 'pyenv-virtualenv`
What I want to automate is:

*

*reboot

*activate my virtual_environment as soon as reboot

*run a command (in this case, honcho start ~~. honcho can be installed via pip) after that
What I tried:

*

*In crontab list (crontab -e)

*Add @reboot sh /home/my_name/launch.sh
launch.sh
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

pyenv activate super_crawler_project
sleep 5

cd /home/my_name/Workspace/super_crawler_project/  # Procfile.dev is located in /home/my_name/Workspace/super_crawler_project/
honcho start -f Procfile.dev $* > /home/my_name/django_log.txt 2>&1

When I reboot, there is an error log left in a django_log.txt:
/home/my_name/launch_django.sh: 8: /home/my_name/launch_django.sh: honcho: not found
But honcho is surely installed in my virtual environment.
However, when I run this command(sh /home/my_name/launch_django.sh) by myself in the Terminal, it works.
I think it can not activate my virtual environment...I don't know why it happened...
Need your advices. Thanks
Edit
I export PATH at the top of launch.sh but still can not find honcho.
django_log.txt
  1 pyenv: honcho: command not found
  2
  3 The `honcho' command exists in these Python versions:
  4   3.6.2/envs/super_crawler_project
  5   super_crawler_project
  6


text:
Q: Is there a way to check if a string contains a date of unknown format? In Python, Is there a way to definitively check if a string contains a date.? It would be preferable if it could catch any date format, but, if not, could it catch the %B-%w-%Y format.

A: You can use dateutil.parser module which will parse any human readable date format for you from a string.
http://labix.org/python-dateutil#head-a23e8ae0a661d77b89dfb3476f85b26f0b30349c


text:
Q: WCF Rest 4 Dependency Injection There are a lot of Dependency Injection examples for Wcf, but many of them deal with versions of Wcf prior to 4.0. I have an app built with Wcf 4 Rest, no .svc file that basically looks like this.
public class RestService : IRestService
{
    IUserRepository _userRepo;

    public RestService(IUserRepository repo)
    {
        _repo = repo;
    }
    public UserModel GetUser(int id)
    {
        return _repo.GetUserById(id);
    }
 }

 public interface IRestService
 {
      [OperationContract]
      [WebInvoke(Method = "GET",
          ResponseFormat = WebMessageFormat.Json,
          UriTemplate = "/User/{id}",
          BodyStyle = WebMessageBodyStyle.Bare)]
      UserModel GetUserById(int id);
 }

Like I said, no svc file, the whole nine yards. Just this in the global.asax file
RouteTable.Routes.Add(new ServiceRoute("RestService", new WebServiceHostFactory(), typeof(RestService)));

So, I am wanting to use Castle Windsor to inject the UserRepository.
What is the best way of setting up the service with this 4.0 implementation to utilize Castle Windsor?

A: Dependency injection in WCF isn't something which can be easily accomplished. You essentially need to implement an IServiceProvider which will be invoked when a new instance of the service needs to be created. You can then use Windsor or some other IoC tool to get the repository instance.
You can find an example of such IoC container in the post about instance providers at http://blogs.msdn.com/b/carlosfigueira/archive/2011/05/31/wcf-extensibility-iinstanceprovider.aspx.


text:
Q: Python Tkinter Validation I am new to Python Tkinter programming. Can someone please help me validate the phone number and email fields? As in the phone number field should accept exactly 10 digits. Email ID should be of the form like in G Mail.
from tkinter import *
import xml.etree.ElementTree as etree
def saveData():
        mobile = etree.SubElement(root,'MOBILE')
        mobile.text = entry_mobile.get()

        email = etree.SubElement(root,'EMAIL')
        email.text = entry_email.get()

        tree=etree.ElementTree(root)
        tree.write("data.xml")
        return

def createWidget():
        global entry_mobile,entry_email
        def testVal(inStr,i,acttyp):
            ind=int(i)
            if acttyp == '1': #insert
                if not inStr[ind].isdigit():
                    return False
            return True
        Label(data, text="DATA").grid(row=0)
        mobile = Label(data, text="Mobile")
        mobile.grid(row=1, sticky=E)
        entry_mobile = Entry(data, bg="powder blue", validate="key")
        entry_mobile['validatecommand'] = (entry_mobile.register(testVal),'%P','%i','%d')
        entry_mobile.grid(row=1, column=1)

        email = Label(data, text="Email ID")
        email.grid(row=2, sticky=E)
        entry_email = Entry(data, bg="powder blue")
        entry_email.grid(row=2, column=1)
        submit = Button(data,text="Save",command = saveData) #binding a function to a widget
        submit.grid(column=1)
        return

data= Tk()
createWidget()
root = etree.Element("DATA")
data.mainloop()

A: Use regular expressions in the validation function.
import re

phonere = re.compile(r'^[0-9]{1,10}$')

The expression ^[0-9]{1,10}$ means that there should be between 1 and 10 digits between the beginning and end of the string.
def is_phone(data):
    return phonere.match(data) != None

If the data matches, the match method returns a SRE_match object. Otherwise it returns None. So if the return of the match method is not none, we have a valid phone number.
vcmd = data.register(is_phone)
entry_mobile['validate'] = 'key'
entry_mobile['validatecommand'] = (vcmd, '%P')

We only need the %P to check the entry field data.
Edit: For e-mail addresses, you could use:
emailre = re.compile(r"(^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$)")

according to this site.

A: All you need to do to validate this is use string manipulation commands. What you need for this is to fit this into separate function inside your code:
if entry_mobile.isnumeric() == True:  #Confirms that all of the characters are numbers
    if entry_email.find('@') != -1:
        if entry_email.find('.') != -1:
            if len(entry_mobile) == 10:
                return True
             else:
                 return False
         else:
             return False
     else:
         return False
 else:
     return False

If it returns true then it is valid by your standards and if it returns false then it is invalid. I hope this helps!


text:
Q: Automatic and periodic zipping (tar.gz) of directory files using shell script I have a system that produces a set of files every day. These files have different formats, but their name is date.
fatemeh@Ubuntu:~ $ls
2020-06-19.all
2020-06-19.log
2020-06-19.ack
2020-06-19.eod
2020-06-20.log
2020-06-20.ack
2020-06-20.eod
2020-06-20.live
2020-06-21.log
2020-06-21.all
2020-06-21.live
2020-06-21.ack

I have to zip the files from three days ago into a file named the same day every day.
I have a script that compresses files in the specified directory by receiving the date.
#! /bin/bash

cd /home/ihubadm/history
####Asking user to enter Date and name######
echo -n "please enter date yy-mm-dd "
read -p 'Date:' datevar

tar -czvf $datevar.tar.gz $datevar.*

echo "Done! $datevar.tar.gz is now available!"

But I want this script to zip three days ago files by reading the current date.
I also tried this one:
datevar=$(date -d + '%Y-%m-%d') && tar -czvf "$datevar.tar.gz" "$datevar".*

but an error raised and it didn't work.
I've read similar pages, but none of them helped solve my problem.
Thank you in advance for your help.


text:
Q: Schema comparison of two dataframes in scala I am trying to write some test cases to validate the data between source (.csv) file and target (hive table). One of the validation is the Structure validation of the table.
I have load the .csv data (using a defined schema) into one dataframe and extracted the hive table data into another dataframe.
When I now try to compare the schema of the two dataframes, it returns false. Not sure why. Any idea on this please?
source dataframe schema:
scala> res39.printSchema
root
 |-- datetime: timestamp (nullable = true)
 |-- load_datetime: timestamp (nullable = true)
 |-- source_bank: string (nullable = true)
 |-- emp_name: string (nullable = true)
 |-- header_row_count: integer (nullable = true)
 |-- emp_hours: double (nullable = true)

target dataframe schema:
scala> targetRawData.printSchema
root
 |-- datetime: timestamp (nullable = true)
 |-- load_datetime: timestamp (nullable = true)
 |-- source_bank: string (nullable = true)
 |-- emp_name: string (nullable = true)
 |-- header_row_count: integer (nullable = true)
 |-- emp_hours: double (nullable = true)

When I compare, it returns false:
scala> res39.schema == targetRawData.schema
res47: Boolean = false

Data in the two dataframes is shown below:
scala> res39.show
+-------------------+-------------------+-----------+--------+----------------+---------+
|           datetime|      load_datetime|source_bank|emp_name|header_row_count|emp_hours|
+-------------------+-------------------+-----------+--------+----------------+---------+
|2017-01-01 01:02:03|2017-01-01 01:02:03|        RBS| Naveen |             100|    15.23|
|2017-03-15 01:02:03|2017-03-15 01:02:03|        RBS| Naveen |             100|   115.78|
|2015-04-02 23:24:25|2015-04-02 23:24:25|        RBS|   Arun |             200|     2.09|
|2010-05-28 12:13:14|2010-05-28 12:13:14|        RBS|   Arun |             100|    30.98|
|2018-06-04 10:11:12|2018-06-04 10:11:12|        XZX|   Arun |             400|     12.0|
+-------------------+-------------------+-----------+--------+----------------+---------+

scala> targetRawData.show
+-------------------+-------------------+-----------+--------+----------------+---------+
|           datetime|      load_datetime|source_bank|emp_name|header_row_count|emp_hours|
+-------------------+-------------------+-----------+--------+----------------+---------+
|2017-01-01 01:02:03|2017-01-01 01:02:03|        RBS|  Naveen|             100|    15.23|
|2017-03-15 01:02:03|2017-03-15 01:02:03|        RBS|  Naveen|             100|   115.78|
|2015-04-02 23:25:25|2015-04-02 23:25:25|        RBS|    Arun|             200|     2.09|
|2010-05-28 12:13:14|2010-05-28 12:13:14|        RBS|    Arun|             100|    30.98|
+-------------------+-------------------+-----------+--------+----------------+---------+

The complete code looks like below:
//import org.apache.spark
import org.apache.spark.sql.hive._
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import org.apache.spark.sql.functions.{to_date, to_timestamp}
import org.apache.spark.sql._
import org.apache.spark.sql.types._
import org.apache.spark.sql.SparkSession
import java.sql.Timestamp
import java.text.SimpleDateFormat
import java.text._
import java.util.Date
import scala.util._
import org.apache.spark.sql.hive.HiveContext

  //val conf = new SparkConf().setAppName("Simple Application")
  //val sc = new SparkContext(conf)
  val hc = new HiveContext(sc)
  val spark: SparkSession = SparkSession.builder().appName("Simple Application").config("spark.master", "local").getOrCreate()

   // set source and target location
    val sourceDataLocation = "hdfs://localhost:9000/source.txt"
    val targetTableName = "TableA"

    // Extract source data
    println("Extracting SAS source data from csv file location " + sourceDataLocation);
    val sqlContext = new org.apache.spark.sql.SQLContext(sc)
    val sourceRawCsvData = sc.textFile(sourceDataLocation)

    println("Extracting target data from hive table " + targetTableName)
    val targetRawData = hc.sql("Select datetime,load_datetime,trim(source_bank) as source_bank,trim(emp_name) as emp_name,header_row_count, emp_hours from " + targetTableName)

    // Add the test cases here
    // Test 2 - Validate the Structure
       val headerColumns = sourceRawCsvData.first().split(",").to[List]
       val schema = TableASchema(headerColumns)

       val data = sourceRawCsvData.mapPartitionsWithIndex((index, element) => if (index == 0) element.drop(1) else element)
       .map(_.split(",").toList)
       .map(row)

       val dataFrame = spark.createDataFrame(data,schema)
       val sourceDataFrame = dataFrame.toDF(dataFrame.columns map(_.toLowerCase): _*)
       data.collect
       data.getClass
    // Test 3 - Validate the data
    // Test 4 - Calculate the average and variance of Int or Dec columns
    // Test 5 - Test 5

  def UpdateResult(tableName: String, returnCode: Int, description: String){
    val insertString = "INSERT INTO TestResult VALUES('" + tableName + "', " + returnCode + ",'" + description + "')"
    val a = hc.sql(insertString)

    }

  def TableASchema(columnName: List[String]): StructType = {
    StructType(
      Seq(
        StructField(name = "datetime", dataType = TimestampType, nullable = true),
        StructField(name = "load_datetime", dataType = TimestampType, nullable = true),
        StructField(name = "source_bank", dataType = StringType, nullable = true),
        StructField(name = "emp_name", dataType = StringType, nullable = true),
        StructField(name = "header_row_count", dataType = IntegerType, nullable = true),
        StructField(name = "emp_hours", dataType = DoubleType, nullable = true)
        )
    )
  }

  def row(line: List[String]): Row = {
       Row(convertToTimestamp(line(0).trim), convertToTimestamp(line(1).trim), line(2).trim, line(3).trim, line(4).toInt, line(5).toDouble)
    }

  def convertToTimestamp(s: String) : Timestamp = s match {
     case "" => null
     case _ => {
        val format = new SimpleDateFormat("ddMMMyyyy:HH:mm:ss")
        Try(new Timestamp(format.parse(s).getTime)) match {
        case Success(t) => t
        case Failure(_) => null
      }
    }
  }

  }

A: I've just had the exact same problem. When you read data from Hive the schema's StructField component will sometimes contain Hive metadata in the field metadata.
You can't see it when printing the schemas because this field is not part of the toString definition.
Here is the solution I've decided to use, I just get a copy of the schema with an empty Metadata before comparing it :
schema.map(_.copy(metadata = Metadata.empty))

A: I've had this issue before and it was caused by differences in the StructField.metadata attribute.  It is almost impossible to identify this out of the box, as simple inspection of the StructField's will only show the name, datatype and nullable values.  My suggestion to debug it would be to compare the metadata of your fields.  Something like this maybe:
res39.schema.zip(targetRawData.schema).foreach{ case (r: StructField, t: StructField) =>
  println(s"Field: ${r.name}\n--| res_meta: ${r.metadata}\n--|target_meta: ${t.metadata}")}

If you want to compare schemas but ignore metadata, then I don't have a great solution.  The best that I've been able to come up with is to iterate over the StructFields and manually remove the metadata, then create a temporary copy of the dataframe without metadata.  So you can do something like this (assuming that df is the dataframe you want to strip of of metadata):
val schemaWithoutMetadata = StructType(df.schema.map{ case f: StructField =>
  StructField(f.name, f.dataType, f.nullable)
})
val tmpDF = spark.sqlContext.createDataFrame(df.rdd, schemaWithoutMetadata)

Then you can either compare the dataframes directly or compare the schemas the way you have been attempting.  I assume that this solution would not be performant, so should only be used on small datasets.

A: Option 1 - StructField.toString
Here is another solution based on observation that the string representation of name + DataType + nullable is unique for each column. As seen here the toString implementation of StructField already supports that rule therefore we can directly use it to compare the columns of different schemas:
import org.apache.spark.sql.types.{StructType, StructField}

val schemaDiff = (s1 :StructType, s2 :StructType) => {
      val s1Keys = s1.map{_.toString}.toSet
      val s2Keys = s2.map{_.toString}.toSet
      val commonKeys =  s1Keys.intersect(s2Keys)

      val diffKeys = s1Keys ++ s2Keys -- commonKeys

      (s1 ++ s2).filter(sf => diffKeys.contains(sf.toString)).toList
}

Notice that the field name is case sensitive hence different column names imply different columns.
The steps:

*

*For each schema generate a set of keys where each key has the format  StructField($name,$dataType,$nullable)

*Get intersection of keys

*Subtract intersection from union of keys, that will give us the keys difference (diffKeys)

*Finally, from both schemas get only the elements that their string representation exists in diffKeys

Option 2 - case class, eq, ==
StructField and StructType are both case classes therefore we expect that the eq method and == operator are both based on a hash generated from the values of their members. You can confirm that by applying the change that @cheseaux pointed out, for example:
val s1 = StructType(res39.schema.map(_.copy(metadata = Metadata.empty)))
val s2 = StructType(targetRawData.schema.map(_.copy(metadata = Metadata.empty)))

s1 == s2 // true

Which is expected since == can be applied between two lists of case classes and returns true only if both lists contain identical items. In the previous case == operator has been applied between two StructType
 objects and consequently between two Seq[StructField] objects as we can see in the constructor definition. As already discussed, the comparison in your case was failing since the value of the metadata differed between schemas.
Attention, the == operator is not safe between schemas if we modify the order of the columns. That is because the list implementation of == considers the order of the items as well. To overcome that obstacle we can safely cast the collection into a set with toSet as we shown above.
Finally, we can take advantage of the above observations and rewrite the first version into the next one:
val schemaDiff = (s1 :StructType, s2 :StructType) => {
      val s1Set = s1.map(_.copy(metadata = Metadata.empty)).toSet
      val s2Set = s2.map(_.copy(metadata = Metadata.empty)).toSet
      val commonItems =  s1Set.intersect(s2Set)

      (s1Set ++ s2Set -- commonItems).toList
}

The performance drawback of the 2nd option is that we need to recreate the StructField item by setting metadata = Metadata.empty.

A: Based on @Derek Kaknes's answer, here's the solution I came up with for comparing schemas, being concerned only about column name, datatype & nullability and indifferent to metadata
import org.apache.spark.sql.DataFrame
import org.apache.spark.sql.types.{DataType, StructField}

def getCleanedSchema(df: DataFrame): Map[String, (DataType, Boolean)] = {
    df.schema.map { (structField: StructField) =>
      structField.name.toLowerCase -> (structField.dataType, structField.nullable)
    }.toMap
  }

// Compare relevant information
def getSchemaDifference(schema1: Map[String, (DataType, Boolean)],
                        schema2: Map[String, (DataType, Boolean)]
                       ): Map[String, (Option[(DataType, Boolean)], Option[(DataType, Boolean)])] = {
  (schema1.keys ++ schema2.keys).
    map(_.toLowerCase).
    toList.distinct.
    flatMap { (columnName: String) =>
      val schema1FieldOpt: Option[(DataType, Boolean)] = schema1.get(columnName)
      val schema2FieldOpt: Option[(DataType, Boolean)] = schema2.get(columnName)

      if (schema1FieldOpt == schema2FieldOpt) None
      else Some(columnName -> (schema1FieldOpt, schema2FieldOpt))
    }.toMap
}

*

*getCleanedSchema method extracts information of interest - column datatype & nullability and returns a map of column name to tuple

*getSchemaDifference method returns a map containing only those columns that differ in the two schemas. If a column is absent in one of the two schemas, then it's corresponding properties would be None

A: val csDf = res39       // any source dataframe
val myDf = targetRawData   // target data frame

val csFields = csDf.schema.fields
val myFields = myDf.schema.fields

val csFieldNameTypeMap = csFields.map(f => f.name -> f.dataType).toMap
val myFieldNameTypemap = myFields.map(f => f.name->f.dataType).toMap

val diffFields = csFields.filter(f =>  csFieldNameTypeMap.get(f.name) != myFieldNameTypemap.get(f.name) ).toList
val diffFieldsMyDf = myFields.filter(f =>  csFieldNameTypeMap.get(f.name) != myFieldNameTypemap.get(f.name) ).toList

'diffFields' and 'diffFieldsMyDf' will give you the fields which have different data type than the other. Similar steps can be done for checking 'nullable' also, just replace 'dataType' with 'nullable'

A: This is a Java level Object comparison problem, you should try with .equals(). This mostly works unless different SourceTypes introduce metadata, nullability issues.


text:
Q: Java 8 DateTime API LocalDateTime I have a DTO class that has a LocalDateTime field in it like the following:
public class MyModel {
    @NotNull(message="This field cannot be null")
    @DateTimeFormat(iso = ISO.DATE_TIME)
    private LocalDateTime startDate;

    ...
}

And I have a Spring MVC Controller that has a Post Mapping like the following:
@PostMapping("/save")
public String save(@Valid MyModel myModel, BindingResult validationResult) {
    if (!validation.hasErrors()) {
        this.myService.save(myModel);
    }
    else {
        ...
        return "myview";
    }
}

Everything works fine, except that if I do not select a date from the User Interface, the "BindingResult" object has

"Property 'startDate' threw exception; nested exception is
  java.lang.NullPointerException"

instead of my custom validation message.