|Paste number 308701:||draft response|
|When:||5 years, 8 months ago|
|Share:||Tweet this! | http://paste.lisp.org/+6M71|
On Tue, Mar 01, 2016 at 01:45:39PM -0800, Markus Unterwaditzer wrote: > I think it would make sense to document a few things for packaging > vdirsyncer. Thanks for asking us about this topic. It seems that you are asking us to stop running your test suite. I'd rather not do that, and below I will say why. I'd like to have a dialogue about this, and possibly come to a compromise. In my opinion, the worst case outcome is that you stop distributing tests in your release tarballs. I think that would have a negative impact on the quality of vdirsyncer in the long run. > ## How to test vdirsyncer > > Many packagers try to run the full testsuite on a vdirsyncer > installation, in order to find bugs with their setup. But: 99% of all > packaging-related bugs are caught by seeing if `vdirsyncer --help` > works correctly. In my experience, '--help' usually exercises a very small part of a code base. Is that not the case for vdirsyncer? Running a (good and up-to-date) test suite does help catch problems with our packaging, especially due to upstream portability bugs, but it also helps upstream by finding more serious bugs. Here are 3 examples of recent bugs that we discovered (sometimes independently of others) by running upstream test suites of *stable* release tarballs. The gnupg bugs manifested on some implementations of a specific architecture and not others. The ilmbase bug manifests on 32-bit Intel compatible only. In each case, the software compiled without error! gnupg: 14 of 35 tests failed https://bugs.gnupg.org/gnupg/issue2229 gnupg: FAIL: gpgtar.test http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22826 openexr / ilmbase: 32-bit floating point rounding problem https://lists.nongnu.org/archive/html/openexr-devel/2015-12/msg00001.html For this reason alone, I don't plan to stop running upstream test suites if I can help it. > > The testsuite is not suitable for packaging tests because: > > * It uses [hypothesis](http://hypothesis.readthedocs.org/) to randomly > generate example input data. Automatic test generation is a really > powerful tool for finding encoding-related bugs, but it also means > that tests may randomly fail. I suggest that it's a bad idea to make the build process of your software non-deterministic (I know you think the tests aren't part of that process, but obviously I disagree). If you want to fuzz vdirsyncer, perhaps that belongs in a separate body of tests. Or, you could add a note to INSTALL instructing packagers to skip these tests. Debian is already building >80% of their packages reproducibly , and in Guix we are creating infrastructure to start treating non-reproducible builds as a bug. I don't see this trend turning around; it's too important for reliability and security. What do you think?  https://tests.reproducible-builds.org/reproducible.html > > * It launches CardDAV/CalDAV servers for integration testing. In other > words, it's not just for finding bugs in vdirsyncer, but also > elsewhere. Why would vdirsyncer-packagers care if a server introduced > a bug that makes the tests fail? I don't see how this is different from bugs being introduced in another one of vdirsyncer's build- or run-time dependencies. I want to know if an update to e.g. 'click' is going to break our vdirsyncer package, or expose a bug in vdirsyncer, etc. Nevertheless, as above, it could go in a separate test suite or come with a caveat in INSTALL. > > * It requires very recent dependencies. Older pytest, hypothesis or > Radicale versions may fail with bizzarre errors. This is especially a > problem if packagers try to use their system-packages for installing > test dependencies instead of just using a virtualenv and pip (I don't > see a reason why that would be a problem TBH) See also #334 Why not just require certain versions in setup.py? You are already doing that for some of your dependencies. > > This seems to be the sanest approach to testing vdirsyncer as a > packager: https://github.com/Homebrew/homebrew/pull/41208 i.e. just > create an example config and run a simple sync. Even a simple > `vdirsyncer --help` catches 80% of errors IMO. > > ## Updating vdirsyncer > > [The Hypothesis Packaging > Guidelines](https://hypothesis.readthedocs.org/en/release/packaging.html) > say that package maintainers who are unable to keep up with its > release cycle are doing the community a disservice. While I think > there are nicer ways to phrase that (especially towards people who are > likewise doing a voluntary service), I am worried that vdirsyncer will > find itself in a similarly problematic situation once [Debian starts > to package > it](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=745027). > Packages usually have been very quickly updated otherwise. Agreed. > > Apart from that we should probably provide an RSS feed for new > releases. GitHub has a tags RSS feed that is mentioned [in the > docs](https://vdirsyncer.readthedocs.org/en/latest/changelog.html), I > guess we could just use that. I prefer an 'announce' mailing list. > > ## Enforcing dependencies' minimum required versions > > Vdirsyncer has a lower bound on the versions of some dependencies. As > far as I understand those lower bounds have been completely ignored so > far by downstream. Perhaps there's just no way to fix this situation. > Again I'm more worried about Debian than any other distro here, but > maybe it'll be fine. Guix uses setup.py when building so we can't ignore your requirements without patching setup.py. Also, it's trivial to have multiple package versions in the distribution (also trivial for users to do this on their own), so I don't see this being an issue unless you end up requiring some version with unpatched vulnerabilities. Respectfully awaiting your response, Leo
This paste has no annotations.