Title: Heya! Not the usual sort of thing to be posted here, but I wanted to show off what I made yesterday. Here&#x27;s a sample page about H1-B visas issued in Bogota:<p>&lt;https:&#x2F;&#x2F;visawhen.com&#x2F;consulates&#x2F;bogota&#x2F;h1b&gt;<p>The code is source-available (not open source) at &lt;https:&#x2F;&#x2F;github.com&#x2F;underyx&#x2F;visawhen&gt;. It&#x27;s my first time choosing a source-available license over MIT, mainly out of fear of existing immigration startups just gobbling this data and code up; frankly I didn&#x27;t think the implications through though, I just threw a safe license on there.<p>The way the project works is:<p>- Use requests-html to find publicly available PDFs from government pages<p>- Use camelot to OCR the PDFs and extract data tables from them<p>- Since the previous step takes crazy long for my tastes (around 8000 pages at around 5 seconds each) I&#x27;ve used dask to split the work into chunks and parallel-process them across my laptop&#x27;s CPUs.<p>- Do data cleanup and processing with pandas, and save all of it to a SQLite file.<p>- Take data from the SQLite file with next.js and generate a static HTML page for each possible embassy - visa type combination<p>- The pages use ECharts to visualize data, and Bulma as a CSS framework<p>- Build and host each commit via Netlify<p>- But proxy to Netlify from CloudFlare, which I believe has more edge locations in the free plan<p>- Collect any donations via Ko-Fi<p>- Use Google Analytics to have a general idea about visitor counts<p>- Use FullStory session recordings to find out about bugs – I&#x27;ve fixed quite a few and I think I&#x27;ll probably remove this tracking after a bit of time<p>…and that&#x27;s where I&#x27;m at now. I&#x27;m pretty happy about the results. Most pages load in less than 300ms, which is something I care about all too much. More importantly, I&#x27;ve shared the site with some immigration communities I&#x27;m part of, and the response has been very positive! Let me know what y&#x27;all think.
Upvote: