They say that "necessity is the mother of invention", well necessity and sometimes an itch to do the same yourself. This is how TorpedoSync came about, from the use of other software to sync "My Documents" across my machines and finding them troubling. Hence the immortal cry "I can do better!".
I start out by using bitsync which then became resilio sync (a free-mium product) which I liked at first but then became a real memory and disk space hog.
After searching a lot, I came across syncthing which is open source and written in go, which I found seemed to be connecting to the internet and was not as nice to use as resilio.
At this point I gave up and decided to write my own.
TorpedoSync
bitsync
resilio sync
syncthing
go
resilio
The source code can be found on github also at :
TorpedoSync has the following features:
node_modules
To run TorpedoSync just execute the file and it will start up and bring up the Web UI in your browser.
Alternatively you can install TorpedoSync as a windows service via torpedosync.exe /i and uninstall it with torpedosync.exe /u
torpedosync.exe /i
torpedosync.exe /u
On the machine that you want to share a folder from, click on the add button under the shares tab:
width="600px" data-src="/KB/IP/1224849/ts_addshare2.jpg" class="lazyload" data-sizes="auto" data->
Enter the required values, and optionally copy one of the tokens you want other machines to connect to.
On the client machines, open the connections tab and click on the connect to button:
width="600px" data-src="/KB/IP/1224849/ts_connectto2.jpg" class="lazyload" data-sizes="auto" data->
Enter the required values and save, the client will wait for confirmation from the master:
width="600px" data-src="/KB/IP/1224849/ts_confirm.jpg" class="lazyload" data-sizes="auto" data->
Click on the confirm button under the connections list and the machine name of the client, after which the client will start syncing.
To see the progress you can click on the connection share name to see the connection information or you can switch to the logs tab.
width="600px" data-src="/KB/IP/1224849/ts_logs2.jpg" class="lazyload" data-sizes="auto" data->
width="600px" data-src="/KB/IP/1224849/ts_cinfo2.jpg" class="lazyload" data-sizes="auto" data->
You can view and change the TorpedoSync configuration via the settings tab
width="600px" data-src="/KB/IP/1224849/ts_settings2.jpg" class="lazyload" data-sizes="auto" data->
Torpedosync takes a different approach to other file sync applications like resilio and syncthing in that it does not compute a cryptographic hash for each file and consequently is much faster and uses less CPU cycles.
Torpedosync
No relational database is used to store the file information, so again the performance is higher and the storage is much smaller.
TorpedoSync only works in a local network environment and does not sync across the internet, so you are potentially much safer, and your data is contained to your own network.
To understand what is implemented in TorpedoSync you must be familiar with the following definitions:
For sync to work, machines on both ends of the connection should be roughly time synchronized, especially when 2 way or read write syncing if performed.
In the case of syncing My Documents between your own machines then this is not very important since you would not be changing files at the same time on both machines. If this does happen then you can recover your files from the .ts\old folder in your share directory.
As stated you have 2 types of synchronization, read only and read/write.
The client in this mode does the following :
State
Delta
.ts\old
The server in this mode does the following:
Once a Delta is generated, deleted files and folders defined in it are moved to .ts\old under the exact folder structure of the original files and the added and changed files are queued for downloading.
The downloader process the queue in one of two ways:
The download block size is configurable and should reflect the bandwidth of the network connection, i.e. for wired networks 10mb should be fine, and for wireless networks 1mb would be better (1 block per second).
The download outcome can be one of the following:
The error queue is no more than a container for failed files and reset, since the files in question will probably be in the next sync anyway.
The source code is composed of the following:
Once a Share is created on the master, TorpedoSync creates a .ts folder in that shared folder for "old archive" files which is named old. Also in the .ts folder there is a file named .ignore which you can use to define what files and folders to ignore in the sync process ( i.e. will not be transferred).
.ts
old
.ignore
Besides the EXE, TorpedoSync creates a log folder for logs and a Computers folder for connection information. Inside the Computers folder a folder is created for each connected machine which will be the machines name, which stores the state, config and download queue for each share that that computer is connected to.
log
Within the share folder inside the special .ts directory you have this special file which contains the list of files and folder you want to ignore when syncing, each line is a ignore item which follows the rules below:
#
* or ?
\
For example:
# comment line
obj # any files and folder under 'obj'
*.suo # any file ending in .suo
code\cache # any files and folder inside 'code\cache'
The settings.config file resides beside the TorpedoSync EXE file and stores the following configuration parameters in a json format:
This file stores the configuration for the share connections in a JSON format:
In 2-way read/write mode, this file stores the last list of files and folders form which TorpedoSync derives which files and folders were deleted since the last sync process.
If this file does not exist (i.e. first time sync), then TorpedoSync assumes that it is the first time and all missing files are transferred to both ends of the connection (no missing files and folders are deleted on both ends of the connection).
This file is in a binary json format.
These files store the files which should be downloaded from the other machine, and the list of failed download files.
These files are in a binary json format.
These files store the share statistics and metrics information, and are in binary json format.
TorpedoSync repurposed some code from RaptorDB which I initially included raptordb.common.dll, but then decided to copy the code to the solution so I get one EXE to deploy which makes things easier for users.
RaptorDB
The project is divided into two parts:
The Web UI is built using vue.js.
vue.js
So you must have the following to compile the source code:
npm install
You can compile the c# code in Visual Studio.
You can debug the Web UI by running WebUI\run.cmd which will startup a node development server for debugging and fast code compilation in webpack.
After you are good with the UI code you do the following :
The following are a highlight of the pain points when writing this project.
While Windows supports long filenames (>260 characters), .net seems to be lacking in this area (especially .net 4). These files especially arise when working with node_modules folders which nest very deep and frequently go over the 260 character limit.
To handle this there are special longfile.cs and longdirectory.cs source files which use the underling OS directly.
One other pain point I encountered was that even Path.GetDirectoryName() suffers from this so I had to write a string parser to handle it.
Path.GetDirectoryName()
My initial UDP code didn't seem to get responses from all other machines in the network, which really took me a long time to figure out, the problem being, I was closing the UDP connection and not getting all the responses in the stream.
So the solution was to stay in a while loop and process not just the first response but any other responses or time out.
while
Now this one took a long long time to figure out, and the problem started when the 2 way sync was downloading files that seemed to have already been downloaded and worst seemed to be going back to the master in a loop.
After much head scratching the problem seemed to be seen when the batch zip feature was used and going through the code by Jaime Olivares I found this gem in the comments :
DOS Date and time:
MS-DOS date. The date is a packed value with the following format. Bits Description
0-4 Day of the month (1..31)
5-8 Month (1 = January, 2 = February, and so on)
9-15 Year offset from 1980 (add 1980 to get actual year)
MS-DOS time. The time is a packed value with the following format. Bits Description
0-4 Second divided by 2
5-10 Minute (0..59)
11-15 Hour (0..23 on a 24-hour clock)
The problem being the seconds are divided by 2 when stored. After much research, I found this was a limitation of FAT in MSDOS back in the day and was carried through to today in the original zip format, later extensions to zip support the full date time, but unfortunately the code from Jaime didn't have this but it did have a comment section for each file zipped and I used that to store the date time for each file until Jaime gets round to supporting the extended zip headers.
I'm not a web developer, and I'm recently getting into it, so figuring out webpack and npm was tricky for me especially the build process, which takes a long time. Also configuring Webpack for special functionality is really hard and frustrating since I always get errors from the snippets I found on the web, so I basically gave up for the most part.
Webpack
After some time I found out that if you npm run dev the code you get hot code changes in your browser and the development cycle is much shorter and nicer (i.e. you don't have to build after each code change and wait and wait...)
npm run dev
The problem with npm run dev is that the web code runs on port 8080 and my server runs on port 88 so I came up with the following workaround in the debug.js file :
debug.js
import './main.js'
window.ServerURL = "";
This sets a variable to my own web server in the c# code, and in the production build I ignore it and just package the main.js file so the webpack.config.js has the following:
var path = require('path')
var webpack = require('webpack')
module.exports = {
entry: './src/debug.js', //normal dev build
...
if (process.env.NODE_ENV === 'production') {
module.exports.entry = './src/main.js' // if production
module.exports.devtool = '#source-map'
...
The only problem I encountered with this approach is that I can't POST json to my server and I get CORS (cross origin resource sharing) errors in the browser and my request don't make it to my server in developer mode.
While I only had 1 point in the code which I posted data here and I could work around it, in a larger project this will be a problem, so a better solution is needed.
The following are a list of what I enjoyed using while working on this project.
At first I used Visual Studio Community 2017 for the Web UI code, but I found that the vue.js support was lacking and not very helpful. So I decided to use Visual Studio Code with the help of some extensions for vue.js, and I have to say that it is phenomenal and very enjoyable.
The general usability and productivity is very high and I absolutely recommend it for web development.
After getting webpack working and support for compiling .vue files in place I have to say that vue.js is outstanding and simple to code by (.vue files are a great help and simplify the development process).
.vue
I shopped around a lot for a "framework" and found vue.js powerful and most importantly simple to learn and use for someone who has never worked in JavaScript. I also looked at angular.js but I found that too complex and difficult.
angular.js
data-src="/KB/IP/1224849/tor_on_pi-r-700.gif" class="lazyload" style="cursor: pointer; border: 0; width: 700px; height: auto" onclick="imageCleanup.showFullImage('/KB/IP/1224849/tor_on_pi.gif')" data-sizes="auto" data->
The refactoring of the code is done and TorpedoSync can now run on the raspberry pi and on the Ubuntu OS. The above animated gif in on my pi b2+ which is syncing data from my development machine running Win 10.
raspberry pi
Ubuntu
To run TorpedoSync on pi and Ubuntu you must install mono with the following command, the EXE file is the same that runs on Windows.
sudo apt-get install mono-complete
I recently came upon npm3 which is a newer version and has the ability to optimize the dependancy for the web ui project and in doing so my node_modules folder size came down from 76Mb to around 56Mb saving me 20Mb.
npm3
So I switched the build and run scripts accordingly (the usage is the same just add a 3 to the end).
With regular notifications from GitHub regarding security vulnerabilities in some javascript libraries in the node modules for TorpedoSync's web UI, I decided to upgrade webpack from v3 to v4, and what a pain that was, the build scripts failed and trying to fix it was beyond me, given that the scripts were mostly scrounged around from various sources and were beyond me in the first place.
Frustrated by the overly complex webpack packager, I searched around and came across the parcel packager (link) which was simple and without configuration and mostly did what I wanted out of the box.
parcel
The only problem I have with parcel is that it appends a hash value to the filename of the outputs, in the documentation it states that this is for the use of content delivery networks, I haven't been able to turn this off and it seems you can't.
This name mangling is a pain when trying to embed the files in TorpedoSync since you have to edit the project file each time it changes.
Other than the above, everything else is wonderful with parcel and I like it a lot since I don't have to fiddle with knobs and configurations.
Continuing my love affair with svelte which started with rewriting the RaptorDB Web UI, I decided to do the same with TorpedoSync. While the structure and mentality of vue and svelte are similar, the process of converting required me to create some svelte components from scratch (I could not find while searching). These components were:
svelte
vue
svelte
The result is quite remarkable :
Some of the highlights:
You can download previous version here:
GetServer.