Paste number 308701: draft response

Paste number 308701: draft response
Pasted by: lfam
When:5 years, 8 months ago
Share:Tweet this! | http://paste.lisp.org/+6M71
Channel:None
Paste contents:
Raw Source | XML | Display As
On Tue, Mar 01, 2016 at 01:45:39PM -0800, Markus Unterwaditzer wrote:
> I think it would make sense to document a few things for packaging
> vdirsyncer.

Thanks for asking us about this topic.

It seems that you are asking us to stop running your test suite. I'd
rather not do that, and below I will say why.

I'd like to have a dialogue about this, and possibly come to a
compromise. In my opinion, the worst case outcome is that you stop
distributing tests in your release tarballs. I think that would have a
negative impact on the quality of vdirsyncer in the long run.

> ## How to test vdirsyncer
> 
> Many packagers try to run the full testsuite on a vdirsyncer
> installation, in order to find bugs with their setup. But: 99% of all
> packaging-related bugs are caught by seeing if `vdirsyncer --help`
> works correctly.

In my experience, '--help' usually exercises a very small part of a code
base. Is that not the case for vdirsyncer?

Running a (good and up-to-date) test suite does help catch problems with
our packaging, especially due to upstream portability bugs, but it also
helps upstream by finding more serious bugs.

Here are 3 examples of recent bugs that we discovered (sometimes
independently of others) by running upstream test suites of *stable*
release tarballs. The gnupg bugs manifested on some implementations of a
specific architecture and not others. The ilmbase bug manifests on
32-bit Intel compatible only. In each case, the software compiled
without error!

gnupg: 14 of 35 tests failed
https://bugs.gnupg.org/gnupg/issue2229

gnupg: FAIL: gpgtar.test
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=22826

openexr / ilmbase: 32-bit floating point rounding problem
https://lists.nongnu.org/archive/html/openexr-devel/2015-12/msg00001.html

For this reason alone, I don't plan to stop running upstream test suites
if I can help it.

> 
> The testsuite is not suitable for packaging tests because:
> 
> * It uses [hypothesis](http://hypothesis.readthedocs.org/) to randomly
> generate example input data. Automatic test generation is a really
> powerful tool for finding encoding-related bugs, but it also means
> that tests may randomly fail.

I suggest that it's a bad idea to make the build process of your
software non-deterministic (I know you think the tests aren't part of
that process, but obviously I disagree). If you want to fuzz vdirsyncer,
perhaps that belongs in a separate body of tests. Or, you could add a
note to INSTALL instructing packagers to skip these tests.

Debian is already building >80% of their packages reproducibly [0], and
in Guix we are creating infrastructure to start treating
non-reproducible builds as a bug. I don't see this trend turning around;
it's too important for reliability and security. What do you think?

[0]
https://tests.reproducible-builds.org/reproducible.html

> 
> * It launches CardDAV/CalDAV servers for integration testing. In other
> words, it's not just for finding bugs in vdirsyncer, but also
> elsewhere. Why would vdirsyncer-packagers care if a server introduced
> a bug that makes the tests fail?

I don't see how this is different from bugs being introduced in another
one of vdirsyncer's build- or run-time dependencies. I want to know if
an update to e.g. 'click' is going to break our vdirsyncer package, or
expose a bug in vdirsyncer, etc. Nevertheless, as above, it could go in
a separate test suite or come with a caveat in INSTALL.

> 
> * It requires very recent dependencies. Older pytest, hypothesis or
> Radicale versions may fail with bizzarre errors. This is especially a
> problem if packagers try to use their system-packages for installing
> test dependencies instead of just using a virtualenv and pip (I don't
> see a reason why that would be a problem TBH) See also #334 

Why not just require certain versions in setup.py? You are already doing
that for some of your dependencies.

> 
> This seems to be the sanest approach to testing vdirsyncer as a
> packager: https://github.com/Homebrew/homebrew/pull/41208 i.e. just
> create an example config and run a simple sync. Even a simple
> `vdirsyncer --help` catches 80% of errors IMO.
> 
> ## Updating vdirsyncer
> 
> [The Hypothesis Packaging
> Guidelines](https://hypothesis.readthedocs.org/en/release/packaging.html)
> say that package maintainers who are unable to keep up with its
> release cycle are doing the community a disservice. While I think
> there are nicer ways to phrase that (especially towards people who are
> likewise doing a voluntary service), I am worried that vdirsyncer will
> find itself in a similarly problematic situation once [Debian starts
> to package
> it](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=745027).
> Packages usually have been very quickly updated otherwise.

Agreed.

> 
> Apart from that we should probably provide an RSS feed for new
> releases. GitHub has a tags RSS feed that is mentioned [in the
> docs](https://vdirsyncer.readthedocs.org/en/latest/changelog.html), I
> guess we could just use that.

I prefer an 'announce' mailing list.

> 
> ## Enforcing dependencies' minimum required versions
> 
> Vdirsyncer has a lower bound on the versions of some dependencies. As
> far as I understand those lower bounds have been completely ignored so
> far by downstream. Perhaps there's just no way to fix this situation.
> Again I'm more worried about Debian than any other distro here, but
> maybe it'll be fine.

Guix uses setup.py when building so we can't ignore your requirements
without patching setup.py. Also, it's trivial to have multiple package
versions in the distribution (also trivial for users to do this on their
own), so I don't see this being an issue unless you end up requiring
some version with unpatched vulnerabilities.

Respectfully awaiting your response,
Leo

This paste has no annotations.

Colorize as:
Show Line Numbers

Lisppaste pastes can be made by anyone at any time. Imagine a fearsomely comprehensive disclaimer of liability. Now fear, comprehensively.