@Allan
I think thats actually correct, when I looked at it at work I was kind of rushed. I think the reason that gzip is an exception is because of the timestamp issue, xz and others don't seem to have that problem.
for those that wanted the source I've pushed it to github:
Offline
There seems to be a problem with the delta for openjdk6:
checking delta integrity... applying deltas... generating openjdk6-6.b17_1.7.2-1-i686.pkg.tar.xz with openjdk6-6.b17_1.7.1-1_to_6.b17_1.7.2-1-i686.delta... xdelta3: non-seekable source in decode: XD3_INTERNAL failed. error: failed to commit transaction (delta patch failed) Errors occurred, no packages were upgraded.
Last edited by Ramses de Norre (2010-04-01 11:27:44)
Offline
Downgrade to 3.0v or 3.0v2. I posted about the issue to xdelta's bugtracker.
Offline
Since there seems to be issues with the newer version of xdelta3. I've uploaded the 3.0v-2 version to archdelta.net
It can be found here:
Offline
Since there seems to be issues with the newer version of xdelta3. I've uploaded the 3.0v-2 version to archdelta.net
It can be found here:
Thanks
And god, this is great on my low speed connection in the students dorm! Thank you a lot for setting up this repo]
60kbps here.. Sucks big time.
Offline
> Sucks big time.
Depends how you see it ;P A phat pipe can suck more.
Offline
Hi
Is it expected behaviour that some packages are not in the repo? (Noticed today: curl and aria2) Is it because maybe they haven't been updated since the repo came into existence?
Offline
If you had posted this on the ML, I would have read this immediately, I barely read the forums nowadays. So I found this thread only now, months after it was created.
But first kudos to all your work, that's awesome !
Ideally we would work together to improve the pacman, repo-add and similar tools, so that all the xdelta code is found either in pacman repo.
And then look at the dbscripts side of things (I know nothing of that though) :
xdelta handles compressed files by uncompressing/recompressing transparently : … ompression
If it does not do that / does not support format compression, it is VERY inefficient in size.
xz support was added relatively recently, Oct 2009, so it might be only present in the last two releases (x and y) :
I did not fully understand why you use the old v release, but I doubt it will work properly with xz.
Now the issue with this external recompression is that pacman (originally/historically) tracks md5sum of *compressed packages*.
This means that when xdelta applies a delta and recompress the final file, that compression must be EXACTLY like the original one. If there is the slightest difference, even just one bit change, pacman will report a corrupted package because of different md5.
The issue with gzip is that by default it applies a timestamp of the time the compression is made. So that clearly break things in our use case.
So we have two things to do :
- disable timestamp at package creation in makepkg : *tar.gz) gzip -f -n "$pkg_file" ;;
- do not let xdelta recompress the file on its own, and apply gzip with -f manually (in above pacman code).
AFAIK there is no timestamp or similar issue with xz, so it might work out of the box.
So the default case is supposed to handle bzip2/xz package.
The move from gzip to xz in repo was not a bad idea, a simple change to use a more efficient compression format, resulting in smaller packages / smaller download for everyone. xz is slower at compression, but reasonably fast at decompression. This seemed ideal : a package is compressed just once (by one developer), but decompressed many times (thousands of users).
Unfortunately this might suck a lot with xdelta :
- the package needs to be recompressed, so the above argument is lost
- xz package are already smaller, and delta ratio is less interesting than with gzip
If we want more flexibility on the recompression part, we might need a big rework of pacman code. We will need to work on uncompressed packages for everything : package integrity with md5, creation and application of delta
I am not sure how big this change would be, and whether it's really worth it.
pacman roulette : pacman -S $(pacman -Slq | LANG=C sort -R | head -n $((RANDOM % 10)))
Offline
I am not sure how big this change would be, and whether it's really worth it.
Well, in my opinion it would be really worth it. It is already a big improvement now, having only to download 2 MB instead of 70 MB when going from jdk-x-1 to jdk-x-2 for example.
Even though, I have to admit, it takes a lot of time now recompressing. Still, you can save much bandwith, on the client and on the server...
Also, isn't it a waste to keep all the old packages one downloads and never use them? (exept for reinstalling things, of course...)
Offline
Unfortunately this might suck a lot with xdelta :
- xz package are already smaller, and delta ratio is less interesting than with gzip
Even though Fedora uses xz, I still usually get a 70%+ reduction in bandwidth (with yum-presto). Just the other day, a 370MB update was reduced to around ±50MB. That is pretty interesting to me.
I admit that it is very possible that I'm misunderstanding.
Offline
@adrianx
I think shining was worried about the complexity of the implementation and smaller benefits wrt to using xdelta3 + xz. Binary deltas give great improvements, so does xz compression, but combining the two might be tricky and
I am not sure how big this change would be, and whether it's really worth it.
Offline
Just to clarify what I said : someone complained that applying delta on xz packages took a long time.
And indeed, that's because applying delta on a compressed package does the following transparently :
1) decompress package (gzip fast, xz fast enough)
2) apply
3) re-compress package (gzip fast, xz slow)
If that's not a problem for you, then good, we already have all that working.
However, if we want to avoid the re-compression step 3) which is slow with xz, it will need a bigger rework than someone might think.
Because we currently do not have any controls on that, it's all done transparently by xdelta3.
pacman roulette : pacman -S $(pacman -Slq | LANG=C sort -R | head -n $((RANDOM % 10)))
Offline
Hi
Is it possible to create locale mirror with using a rsync or something else?
Offline
I set up my own delta mirror, for x86_64:
So far it hasn't generated any deltas, since I'm too lazy to use ARM. :-P
Offline
This is great, it's been saving me lots of time on my excruciatingly slow connection.
I've noticed that the latest updates to wesnoth (1.8.2-2) and xulrunner (1.9.2.4-1) don't seem to be available. Especially wesnoth would be really useful since it's just a pkgrel bump for a boost rebuild, and 260mb download.
Does this mean the deltas on these didn't meet the 70% rule? Or is there some other reasons for leaving some packages out?
Offline
It seems like the repo is not syncing any longer?
Offline
It seems like the repo is not syncing any longer?
Just noticed this post.. yeah you're right.. apparently my python and openssl got out of sync so I was getting the following error:
import md5 File "/usr/lib/python2.6/md5.py", line 10, in <module> from hashlib import md5 File "/usr/lib/python2.6/hashlib.py", line 136, in <module> md5 = __get_builtin_constructor('md5') File "/usr/lib/python2.6/hashlib.py", line 63, in __get_builtin_constructor import _md5 ImportError: No module named _md5
Anyways, doing a pacman -Syu.. I'll leave it overnight since it's a huge upgrade:
Total Installed Size: 5840.26 MB
As long as nothing blows up on my desktop, I'll reboot tomorrow and hopefully everything will start syncing again properly.
Wish I had a server to run this on.. t's running on my desktop currently and.. well.. desktop isn't exactly the most stable machine in the world...
Thanks for bringing this up
Offline
Maybe you could post some stats (size of that thing) and ask for a bit of server space / bandwidth?
Offline
Maybe you could post some stats (size of that thing) and ask for a bit of server space / bandwidth?
8.3G community/os/i686 368M community-testing/os/i686 259M core/os/i686 9.0G extra/os/i686 1.9G testing/os/i686 407M community/os/i686/deltas 8.6M community-testing/os/i686/deltas 64M core/os/i686/deltas 1.5G extra/os/i686/deltas 87M testing/os/i686/deltas 22G total
The scripts are pretty small in size, however a full copy of the repository needs to be accessible to use for generating the diffs.
I wish I could give this project more time, but work killed the hobby side of IT for me
Anyways, if someone has good bandwith 24/7 uptime, and 30gigs (60gigs if they wanna do x64 also) free and wants to run this locally.. I'll be glad to help them get set up.
Also.. looking at the above, its only 2gigs (4gigs if x64 also) or so for the delta only portions (thats all a webserver would need to serve)
I'm sorry for not supporting this as well as I could.. Having Linux work be my day job really killed the hobby side for me
.
Hopefully someone takes the code and improves on this project or it gets integrated into the official repos.
Offline
> Having Linux work be my day job really killed the hobby side for me
Many people would be happy to work w/ Linux:
> if someone has good bandwith 24/7 uptime
How much bandwidth is needed /month?
Were you creating deltas for all the packages? I don't see wesnoth and IIRC there were also some other big packages missing.
sandsmark provides 64-bit already
Offline
> Having Linux work be my day job really killed the hobby side for me
Many people would be happy to work w/ Linux:
Don't get me wrong, I love my job. That said, 40hrs/week is enough Linux for me. The "hobby" (fun little project here and there) died for the most part. I still love Linux.
> if someone has good bandwith 24/7 uptime
How much bandwidth is needed /month?
Hmm.. actually the 24/7 uptime is only needed on the server hosting the deltas. The delta creating machine only needs to sync, create deltas, and then upload (I do this nightly, but others may prefer more/less).
Were you creating deltas for all the packages? I don't see wesnoth and IIRC there were also some other big packages missing.
As for wesnoth and (possibly) other missing packages:
creating delta: wesnoth-1.8.2-1-i686.pkg.tar.xz -> wesnoth-1.8.2-2-i686.pkg.tar.xz => wesnoth-1.8.2-1_to_1.8.2-2-i686.delta Delta size too big: /var/repos/extra/os/i686/deltas/wesnoth-1.8.2-1_to_1.8.2-2-i686.delta deleting /var/repos/extra/os/i686/deltas/wesnoth-1.8.2-1_to_1.8.2-2-i686.delta deleting /var/repos/extra/os/i686/wesnoth-1.8.2-1-i686.pkg.tar.xz
If one delta file is bigger than 70% that leaves 0 deltas.
From my first post:
* I am only keeping up to 3 versions back for each package. Also, delta size combined must be less than %70 of the pkg size which is the pacman cutoff.
sandsmark provides 64-bit already
sandsmark,
Do you have the space/bandwidth to also do i686? If you decide to do so, let me know so I can change archdelta.net to mirror you rather than generating different sets of deltas (pretty sure md5sums will be different).
Offline
If sandsmark doesn't respond, contact him directly:
> If one delta file is bigger than 70% that leaves 0 deltas.
<facepalm> I'm braindead sometimes, sorry.
Offline
Hi all,
xdelta3 author here, just following up on the bug reported in 3.0w, which was fixed in 3.0y. I'm trying to close issue 110, here:
I couldn't determine whether the error was caused by applying pre-3.0w-generated deltas with a 3.0y binary, or whether it was a bug in 3.0y that still exists. In any case, 3.0z is released and I'd appreciate your help with any information. The newer code is faster than 3.0v in the way it handles compressed source files, it decompresses them through a pipe instead of making an uncompressed copy in a temporary file.
And by the way, on the subject of "bsdiff wins.. no surprise there", I wonder if that's a fair comparison. If you're running xdelta3 with its default settings, you're getting only 64MB of source buffer, meaning xdelta3 will not find any similarities that occur at a distance of greater than 64MB. With -B set to the size of the source file you should see an improvement, and you might also set -9 and -S djw if you really want a fair comparison to bsdiff. Thanks, Josh
Offline