<repo_name>godblessforhimself/ThuSearch<file_sep>/readme.md
# 校园搜索引擎

## 资源抓取

### Heritrix3.2.0配置：

#### Seed：使用尽可能多的种子
```
    <prop key="seeds.textSource.value">
# URLS HERE
http://news.tsinghua.edu.cn
http://www.cs.tsinghua.edu.cn/
http://graduate.tsinghua.edu.cn
http://member.artmuseum.tsinghua.edu.cn
http://search.tsinghua.edu.cn
http://ticket.artmuseum.tsinghua.edu.cn
http://tv.tsinghua.edu.cn
http://www.artmuseum.tsinghua.edu.cn
http://www.tsinghua.edu.cn
http://www.tuef.tsinghua.edu.cn
http://yz.tsinghua.edu.cn
http://zhaopin.rsc.tsinghua.edu.cn
http://student.tsinghua.edu.cn
http://www.dangjian.tsinghua.edu.cn
http://academic.tsinghua.edu.cn
http://myhome.tsinghua.edu.cn
http://career.tsinghua.edu.cn
http://pt.tsinghua.edu.cn
http://kyybgxx.cic.tsinghua.edu.cn/kybg/index.jsp
http://guofang.tsinghua.edu.cn
http://info.itc.tsinghua.edu.cn
http://thdag.cic.tsinghua.edu.cn
http://xsg.tsinghua.edu.cn
http://info.tsinghua.edu.cn
    </prop>
```
#### SurtsSource
```
<property name="surtsSource">
        <bean class="org.archive.spring.ConfigString">
         <property name="value">
          <value>
           # example.com
           # http://www.example.edu/path1/
           +http://(cn,edu,tsinghua,
          </value>
         </property> 
        </bean>
       </property>
 </bean>
 ```
 #### MatchesListRegexDecideRule:过滤掉非pdf,doc,html和图书馆的链接
 ```
 <bean class="org.archive.modules.deciderules.MatchesListRegexDecideRule">
          <property name="decision" value="REJECT"/>
      <property name="listLogicalOr" value="true" /> 
      <property name="regexList">
           <list>
                <value>^(?:(?!\.html)(?!\.htm)(?!\.pdf)(?!\.PDF)(?!\.doc)(?!\.docx)(?!\.DOC)(?!\.DOCX)(?!\.htmlx)(?!\.HTML)(?!\.HTM)(?!\.HTMLX).)+</value>
                <value>[\S]*lib.tsinghua.edu.cn[\S]*</value>
           <value>[\S]*166.111.120.[\S]*</value>
           </list>
          </property>
    </bean>
 ```
 #### 将默认的warcWriter改为MirorWritor
 ```
  <bean id="warcWriter" class="org.archive.modules.writer.MirrorWriterProcessor"> </bean>
 ```
 #### 资源抓取遇到的困难
   - 按照作业说明提供的1.x版本heritrix的配置抓取不到文档。
   - heritrix 3.x 配置只能通过修改crawl-beans.cxml文档。在建立heritrix 3.x源码版本，建立自己的抓取项目时遇到了一些依赖和maven配置的问题，暂未解决。
   - heritrix 3.x 抓取文档需要2-3天时间。
 ### 参考 https://github.com/ChenGuangbinTHU/THUSearchEngine/blob/master/WebPreprocess/crawler-beans.cxml
 
 ## 资源预处理
 
 ### 总体流程：抓取的资源在mirror目录中。遍历mirror目录，根据文件后缀(doc,pdf,html) ，调用不同的解析方法。解析出有效文本，进行格式处理后保存到mysql中。
      通过访问处理后的文本，生成三个xml文件，用于indexing步骤。
 #### fileParser.java 进行目录遍历，解析文本，存入mysql的java代码
   - main：Files.walkFileTree进行遍历目录，对于每个文件，调用fileParser p.parse(filename)
   - fileParser.parse(String path): 根据path的后缀，调用runPDFParser(path)\runDocParser(path)\runHtmlParser(path)或直接返回;
   - fileParser.runPDFParser(String path): 使用apache PDFBox解析PDF格式文件，将该pdf对应的url,content,type存入mysql ;
   - fileParser.runDocParser(String path): 根据doc文件的版本，分别调用POI库中不同的解析方法获得word文档的文本内容。将word文档的url,content,type存入mysql。
   - fileParser.runHtmlParser(String path): 使用Jsoup提取html中的title,h1,h2,a href内容。打开文档前，用codec(path)判断编码。提取前，用正则表达式过滤掉html中的注释内容。content：getChinese获取html中的中文部分。将url,title,h1,h2,content,outlinks，type存入mysql。title,h1,h2作了长度限制。
      
#### MySql.java 提供了mysql的接口，main负责读取数据库，pagerank的图和映射文件的生成，pagerank的计算，indexing需要的xml文件的生成。
   - MySql(): 初始化时与SchoolSearch数据库建立连接，然后初始化statement。
   - createTables(): 创建pdf,doc,html三个表
   - DropAllTables(): 删除pdf,doc,html三个表
   - update(String sql):用语句sql更新数据库。
   - buildMap_Graph(): 遍历mysql schoolsearch.html,根据每项的url,outlinks建立mapping.txt和graph.txt文件。
   - calculate_pagerank(): 根据mapping.txt graph.txt，进行pagerank计算，写入pagerank.txt文件。
   - write2xml(): 读取schoolsearch html，pdf，doc三个表，分别生成三个xml文件
    
#### 资源预处理遇到的问题
  - Jsoup存在bug，解析html时：解析了注释的代码；返回正文时，把正文和javascript代码一起返回了。
  - 所以预处理时，先过滤了注释的内容，content不直接通过Jsoup获取，而是手动解析中文部分。
  - 编码问题：一部分网页的编码和<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />中设置的不同。如academic.tsinghua.edu.cn/index.html。实际上编码是gbk
    通过使用fileParser.codec(String filename)->fileParser.isUtf(String filename)先判断编码，再用文件流指定编码读入文件。
  - 抓取的一些pdf\doc文档损坏。暂时不记录这些损坏文件的数据。未解决。
  - 抓取的一些doc文件缺失一些css,无法通过POI解析。暂未解决。
  - 遍历一次mirror/下的文件，约3小时。可以并行提速。
  
 
 
 
 
 
<file_sep>/search_engine/fileParser/MySql.java
import org.apache.commons.lang3.StringEscapeUtils;

import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.io.OutputStreamWriter;
import java.net.UnknownHostException;
import java.sql.*;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class MySql {
    static final String JDBC_DRIVER = "com.mysql.jdbc.Driver";
    static final String DB_URL = "jdbc:mysql://localhost:3306/SchoolSearch?characterEncoding=UTF-8&useSSL=false";
    static final String USER = "root";
    static final String PASS = "";
    Connection conn = null;
    Statement stmt = null;
	public PreparedStatement p_insertpdf = null;
	public PreparedStatement p_insertdoc = null;
	public PreparedStatement p_insert8 = null;
	public PreparedStatement p_selectpdf = null;
	public PreparedStatement p_selectdoc = null;
	public PreparedStatement p_selecthtml = null;
    public static String[] tableName = {"pdf", "doc", "html"};
	MySql() 
	{
		getConnection();
		getStatement();
		System.out.println("Database initalize finish!");
    }
	public void getConnection() {
		try {
            Class.forName("com.mysql.jdbc.Driver");
            System.out.println("连接数据库...");
            conn = DriverManager.getConnection(DB_URL, USER, PASS);
        } catch(ClassNotFoundException e){
            System.out.println("加载数据库驱动失败:\n"+e.toString());
            return;
        }
        catch (SQLException e) {
            System.out.println("获取连接失败" + e.toString());
            return;
        }
    }
	public void getStatement() 
	{
        try {
            stmt = conn.createStatement();
            p_insertpdf = conn.prepareStatement("INSERT INTO pdf VALUES (?,?,?);");
            p_insertdoc = conn.prepareStatement("INSERT INTO doc VALUES (?,?,?);");
            p_insert8 = conn.prepareStatement("INSERT INTO html VALUES (?,?,?,?,?,?,?,?);");
			p_selectpdf = conn.prepareStatement("SELECT * FROM pdf;");
			p_selecthtml = conn.prepareStatement("SELECT * FROM html;",ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY);
			p_selecthtml.setFetchDirection(ResultSet.FETCH_REVERSE); 
			p_selecthtml.setFetchSize(Integer.MIN_VALUE);
			p_selectdoc = conn.prepareStatement("SELECT * FROM doc;");
        }catch (SQLException e) {
            System.out.println("getStatement failed " + e.toString());
        }

    }
	public void createTables()
	{
		try{
			String sql = "";
			stmt.executeUpdate("CREATE DATABASE IF NOT EXISTS SchoolSearch character set utf8mb4;");
			sql = "CREATE TABLE IF NOT EXISTS pdf " +
					"(url VARCHAR(255) not null primary key, " +
					"type VARCHAR(4), " +
					"content MEDIUMTEXT) DEFAULT CHARSET=utf8mb4";
			stmt.executeUpdate(sql);
			sql = "CREATE TABLE IF NOT EXISTS doc " +
					"(url VARCHAR(255) not null primary key, " +
					"type VARCHAR(4), " +
					"content MEDIUMTEXT) DEFAULT CHARSET=utf8mb4";
			stmt.executeUpdate(sql);
			sql = "CREATE TABLE IF NOT EXISTS html " +
					"(url VARCHAR(255) not null primary key, " +
					"type VARCHAR(4), " +
					"title VARCHAR(255), " +
					"h1 VARCHAR(255), " + 
					"h2 VARCHAR(255), " +
					"content MEDIUMTEXT, " +
					"outlinks TEXT, " + 
					"pagerank FLOAT) DEFAULT CHARSET=utf8mb4";
			stmt.executeUpdate(sql);
		}
		catch (SQLException se)
		{
			se.printStackTrace();
		}
	}
	public void DropAllTables()
	{
		try {
			String sql = "DROP TABLE pdf;";
			stmt.executeUpdate(sql);
			sql = "DROP TABLE html;";
			stmt.executeUpdate(sql);
			sql = "DROP TABLE doc;";
			stmt.executeUpdate(sql);
		} catch (SQLException e) {
			e.printStackTrace();
		}
	}
    public ResultSet read(String sqlSelect){
        ResultSet rst = null;
        try{
            rst = stmt.executeQuery(sqlSelect);
            System.out.println("查询成功");
        }
        catch(SQLException e){
            System.out.println(e.toString());
        }
        return rst;
    }
    public int update(String sqlUpdate){
        int nRecord = 0;
        try{
            nRecord = stmt.executeUpdate(sqlUpdate);
        }
        catch(SQLException e){
            System.out.println(e.toString());
        }
        return nRecord;
    }
    public Map<String,Double> getPR()
    {
    	Map<String,Double> m = new HashMap<String,Double>();
    	long t0 = System.currentTimeMillis();
    	try{
			BufferedReader in = new BufferedReader(new InputStreamReader(new FileInputStream("pagerank.txt"),"utf-8"));
			String line="";
			while ((line = in.readLine()) != null)
			{
				//System.out.println(line);
				//Thread.sleep(10000);
				int p1 = line.indexOf(" ");
				int p2 = line.indexOf(" ", p1 + 1);
				String url = line.substring(p2 + 1);
				Double pagerank = Double.valueOf(line.substring(p1 + 1, p2));
				//System.out.println("url="+url + " pagerank = " +pagerank);
				
				m.put(url, pagerank);
			}
			in.close();
			System.out.println("get pagerank: time " + (System.currentTimeMillis() - t0)/1000 + " seconds");
		}catch (Exception e){
			e.printStackTrace();
		}
    	return m;
    }
    public void closeAll(){
        try{
			stmt.close();
			conn.close();
        }
        catch(SQLException e){
            System.out.println(e.toString());
            return;
        }
    }
	public static String processString(String s) 
	{
        s = s.replace( "&", "&amp;" );
		s = s.replace( "<", "&lt;" );
		s = s.replace( ">", "&gt;" );
		s = s.replace( "\"", "&quot;" );
		s = s.replace( "\'", "&apos;" );
		s = s.replaceAll("[\\000]+", "");
		s = s.replaceAll("\\s{2,}","\n");
		StringBuffer out = new StringBuffer(); // Used to hold the output.
		char current; // Used to reference the current character.
		if (s == null || ("".equals(s))) 
			return ""; // vacancy test.
		for (int i = 0; i < s.length(); i++) 
		{
			current = s.charAt(i); // NOTE: No IndexOutOfBoundsException caught here; it should not happen.
			if ((current == 0x9) ||
					(current == 0xA) ||
					(current == 0xD) ||
					((current >= 0x20) && (current <= 0xD7FF)) ||
					((current >= 0xE000) && (current <= 0xFFFD)) ||
					((current >= 0x10000) && (current <= 0x10FFFF)))
				out.append(current);
		}
		return out.toString();
	}
	public BufferedWriter write2file(String filename)
	{
		File file = new File(filename);
		try{
			if (!file.exists())
				file.createNewFile();
			OutputStream fw=new FileOutputStream(file);
			BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(fw,"utf-8"));
			bw.write("<?xml version=\"1.0\" encoding=\"utf-8\"?>\n");
			bw.write("<pics>\n" +
					"\t<category name=\""+filename.replace(".xml", "")+"\">\n");
			return bw;
		}catch (Exception e)
		{
			e.printStackTrace();
		}
		return null;
	}
	public void end(BufferedWriter bw)
	{
		try {
			bw.write("</category>\n" +
			        "</pics>");
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
        try {
			bw.close();
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
	}
	public void write2xml()
	{
		String f1 = "SchoolSearch_pdf.xml";
		String f2 = "SchoolSearch_doc.xml";
		String f3 = "SchoolSearch_html.xml";
		try{
			long t0 = System.currentTimeMillis();
			int cnt = 0;
			ResultSet rst = p_selectpdf.executeQuery();
			BufferedWriter bw = write2file(f1);
	        while (rst.next()) 
			{
	            String url = processString(rst.getString("url"));
	            String content = processString(rst.getString("content"));
	            bw.write("<pic url=\""+url+"\" type=\"pdf\" content=\""+content+ "\" />\n");
				cnt += 1;
	        }
	        end(bw);
	        bw = write2file(f2);
	        rst = p_selectdoc.executeQuery();
	        while (rst.next()) 
			{
	            String url = processString(rst.getString("url"));
	            String content = processString(rst.getString("content"));
	            bw.write("<pic url=\"" + url + "\" type=\"doc\" content=\"" + content + "\" />\n");
				cnt += 1;
	        }
			end(bw);
			rst = p_selecthtml.executeQuery();
			Map<String,Double> pagerank = getPR();
			t0 = System.currentTimeMillis();
			bw = write2file(f3);
			while (rst.next())
			{
	            String url = rst.getString("url");
	            String content = processString(rst.getString("content"));
				String title = processString(rst.getString("title"));
				String h1 = processString(rst.getString("h1"));
				String h2 = processString(rst.getString("h2"));
				String anchor = processString(rst.getString("outlinks"));
				Double pr = pagerank.get(url);
				url = processString(url);
				if (pr == null)
				{
					System.out.println(url + ": pagerank not found");
					pr = 1E-7;
				}
				bw.write("<pic url=\""+url+"\" type=\"html\" title=\""+title+"\" h1=\""+h1+"\" h2=\""+h2+"\" content=\""+content+ "\" anchor=\"" + anchor + "\" pagerank=\""+String.valueOf(pr)+"\" />\n");
				cnt += 1;
				if (cnt % 10000 == 0)
				{
					System.out.println(cnt+":time " + (System.currentTimeMillis() - t0) / 1000 + " seconds");
				}
	        }
			end(bw);
		}catch (Exception e)
		{
			e.printStackTrace();
		}
	}
	public static Map<Integer,String> loadMap()
	{
		Map<Integer,String> m = new HashMap<Integer,String>();
		try{
			BufferedReader in = new BufferedReader(new InputStreamReader(new FileInputStream("mapping.txt"),"utf-8"));
			String line="";
			while ((line = in.readLine()) != null)
			{
				String part[] = line.split(":");
				Integer src = Integer.valueOf(part[0]);
				m.put(src,part[1]);
			}
			in.close();
		}catch (Exception e){
			e.printStackTrace();
		}
		return m;
	}
	public static ArrayList<ArrayList> loadEdge()
	{
		ArrayList<ArrayList> m = new ArrayList<ArrayList>();
		try{
			BufferedReader in = new BufferedReader(new InputStreamReader(new FileInputStream("graph.txt"),"utf-8"));
			String line="";
			while ((line = in.readLine()) != null)
			{
				ArrayList<Integer> a = new ArrayList<Integer>();
				String part[] = line.split("->");
				Integer src = Integer.valueOf(part[0]);
				a.add(src);
				if (part.length==1)
					continue;
				String[] dsts = part[1].split(",");
				for (String dst: dsts)
				{
					a.add(Integer.valueOf(dst));
				}
				if (a.size() > 1)
				{
					m.add(a);
					//System.out.println(a.toString());
				}
			}
			in.close();
		}catch (Exception e){
			e.printStackTrace();
		}
		return m;
	}
	public void buildMap_Graph()
	{
		try
		{
			long t0 = System.currentTimeMillis();
			PreparedStatement p = conn.prepareStatement("SELECT url,outlinks FROM html;");
			ResultSet rst = p.executeQuery();
			System.out.println("Query use " + (System.currentTimeMillis() - t0)/1000 + " seconds");
			t0=System.currentTimeMillis();
			
			Map<String,Integer> map = new HashMap<String,Integer>();
			ArrayList<String> indexing = new ArrayList<String>();
			
			File graph = new File("graph.txt");
			if (!graph.exists())
				graph.createNewFile();
			OutputStream fw=new FileOutputStream(graph);
			BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(fw,"utf-8"));
			int count = 0;
			int loop = 0;
			while (rst.next())
			{
				String forGraph = "";
				String url = rst.getString("url");
				Integer mapindex;
				if ((mapindex = map.get(url)) != null)
				{
					forGraph += Integer.toString(mapindex) + "->";
				}
				else
				{
					map.put(url, count);
					indexing.add(url);
					forGraph += Integer.toString(count) + "->";
					count += 1;
				}
				String outlinks = rst.getString("outlinks");
				if (outlinks.equals(""))
					continue;
				String links[] = outlinks.split("\n");
				for (String link: links)
				{
					if (link.equals(""))
						continue;
					int towrite = 0;
					Integer i = map.get(link);
					if (i == null)
					{
						map.put(link, count);
						indexing.add(link);
						towrite = count;
						count += 1;
					}
					else
					{
						towrite = i;
					}
					if (forGraph.endsWith("->"))
					{
						forGraph += Integer.toString(towrite);
					}
					else
					{
						forGraph += "," + Integer.toString(towrite);
					}
				}
				bw.write(forGraph + "\n");
				loop += 1;
				/*if (loop % 5000 == 0)
				{
					System.out.println(loop + ":time passed " + (System.currentTimeMillis() - t0) / 1000 + " seconds");
				}*/
			}
			p.close();
			rst.close();
			bw.close();
			fw.close();
			File mapping = new File("mapping.txt");
			if (!mapping.exists())
				mapping.createNewFile();
			fw=new FileOutputStream(mapping);
			bw = new BufferedWriter(new OutputStreamWriter(fw,"utf-8"));
			for (int i = 0; i < indexing.size(); i ++)
			{
				bw.write(Integer.toString(i) + ":" + indexing.get(i) + "\n");
			}
			bw.close();
			fw.close();
			System.out.println("build graph and map use " + (System.currentTimeMillis() - t0) / 1000 + " seconds");
		}catch (Exception e)
		{
			e.printStackTrace();
		}
	}
	public void calculate_pagerank()
	{
		try{
			double alpha = 0.15;
			int TN = 30;
			long t0 = System.currentTimeMillis();
			Map<Integer,String> mapping = loadMap();
			int N = mapping.size();
			int[] outdegrees = new int[N];
			for (int i = 0; i < N; i ++)
			{
				outdegrees[i] = 0;
			}
			double[] PRs = new double[N];
			double[] I = new double[N];
			double S = 0;
			ArrayList<ArrayList> edges = loadEdge();
			for (int i = 0, n = edges.size(); i < n; i ++)
			{
				ArrayList<Integer> edge = edges.get(i);
				Integer src = edge.get(0);
				int outdegree = edge.size() - 1;
				outdegrees[src] = outdegree;
			}
			for (int i = 0; i < N; i ++)
			{
				PRs[i] = 1 / (double)N;
				I[i] = alpha / (double) N;
				if (outdegrees[i] == 0)
				{
					S += PRs[i];
				}
			}
			System.out.println("time " + (System.currentTimeMillis() - t0) / 1000 + " seconds");
			t0 = System.currentTimeMillis();
			for (int k = 1; k <= TN; k ++)
			{
				for (int i = 0, n = edges.size(); i < n; i ++)
				{
					ArrayList<Integer> edge = edges.get(i);
					Integer src = edge.get(0);
					int outdegree = edge.size() - 1;
					for (int j = 0; j < outdegree; j ++)
					{
						Integer dst = edge.get(j + 1);
						I[dst] += (1 - alpha) * PRs[src] / (double) outdegrees[src];
					}
				}
				S = 0.0;
				for (int i = 0; i < N; i ++)
				{
					PRs[i] = I[i] + (1 - alpha) * S / N;
					I[i] = alpha / (double) N;
					if (outdegrees[i] == 0)
					{
						S += PRs[i];
					}
				}
				//System.out.println("k = " + k + " time " + (System.currentTimeMillis() - t0) / 1000 + " seconds");
			}
			File pagerank = new File("pagerank.txt");
			if (!pagerank.exists())
				pagerank.createNewFile();
			OutputStream fw=new FileOutputStream(pagerank);
			BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(fw,"utf-8"));
			for (int i = 0; i < N; i ++)
			{
				bw.write(Integer.toString(i) + " " + Double.toString(PRs[i]) + " " + mapping.get(i) + "\n");
			}
			bw.close();
			fw.close();
			System.out.println("calculating pagerank use " + (System.currentTimeMillis() - t0) / 1000 + " seconds");
		}catch (Exception e){
			e.printStackTrace();
		}
	}
	public void updatePR2DB()
	{
		try{
			BufferedReader in = new BufferedReader(new InputStreamReader(new FileInputStream("pagerank.txt"),"utf-8"));
			String str = "";
			PreparedStatement p = conn.prepareStatement("UPDATE html SET pagerank = ? WHERE url = ?;");
			long t0 = System.currentTimeMillis();
			int cnt = 0;
			while ((str = in.readLine()) != null)
			{
				String[] part = str.split(" ");
				String url = part[2];
				double d = Double.valueOf(part[1]);
				float pagerank = (float)d;
				p.setFloat(1, pagerank);
				p.setString(2, url);
				p.executeUpdate();
				cnt += 1;
				if (cnt % 500 == 0)
				{
					System.out.println(cnt + ":use " + (System.currentTimeMillis() - t0)/1000 + " seconds");
				}
			}
			System.out.println("updatePR2DB use " + (System.currentTimeMillis() - t0)/1000 + " seconds");
			in.close();
		}catch (Exception e)
		{
			e.printStackTrace();
		}
	}
	public static void main(String[] args) throws SQLException {
        MySql ms = new MySql();
		//ms.buildMap_Graph();
        //ms.calculate_pagerank();
		ms.write2xml();
        
	}
}