18:08:04 <vagrantc> #startmeeting
18:08:04 <MeetBot> Meeting started Mon Nov 16 18:08:04 2020 UTC.  The chair is vagrantc. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:08:04 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
18:08:39 <vagrantc> #link https://pad.sfconservancy.org/p/reproducible-builds-debugging
18:08:50 <vagrantc> #topic who is here?
18:08:58 * vagrantc = Vagrant Cascadian
18:09:07 * bmwiedemann = Bernhard M. Wiedemann
18:10:43 <bmwiedemann> anyone else? Foxboron? anthraxx?
18:12:14 <Foxboron> Ohno, I forgot about the meeting! Yes I am :(
18:12:18 <vagrantc> i failed to do any reminder emails and whatnot ...
18:12:19 <Foxboron> :)*
18:12:21 <vagrantc> yay
18:13:22 <vagrantc> #topic agenda on the fly
18:14:02 <vagrantc> i could probably walk through my typical steps when debugging reproducibility for a package
18:14:24 <vagrantc> it would be interesting to do for each distro, maybe?
18:14:50 <bmwiedemann> I guess, the general steps of find, debug, fix, submit are the same?
18:15:08 <Foxboron> Im actually curious about your tooling these days. Especially for distribution packages
18:15:56 <vagrantc> bmwiedemann: in general, sure
18:17:38 <bmwiedemann> vagrantc: so please tell us about the details
18:19:30 <vagrantc> ok, i'll talk about debian for a bit
18:19:40 <vagrantc> #debugging Debian packages
18:19:44 <vagrantc> er,
18:19:48 <vagrantc> #topic debugging Debian packages
18:20:44 <vagrantc> so, first thing i do is use https://tests.reproducible-builds.org/debian/PACKAGE
18:21:08 <vagrantc> to see if there is useful build information from the test infrastructure :)
18:21:34 <vagrantc> sometimes i'll see something that only fails on 32-bit architectures, sometimes only on arm* or x86*
18:22:51 <bmwiedemann> how much do you look through the diffoscope output?
18:22:57 <vagrantc> on a good day, it may even come with some diffoscope output
18:23:10 <vagrantc> i definitely scroll through it.
18:23:33 <vagrantc> i'll also look at the differences between unstable and bullseye (our current testing suite)
18:24:10 <bmwiedemann> because one varies build-path and the other doesnt
18:24:22 <bmwiedemann> or because of toolchain patches?
18:24:23 <vagrantc> since we do more variations in unstable and experimental (e.g. build path, and fixfilepath a.k.a. -ffile-prefix-map)
18:25:28 <vagrantc> i like to target non-build-path variations first, especially if i can get rid of all of those in the packages, then the diff becomes smaller eventually :)
18:26:40 <vagrantc> locally, i mostly use reprotest to compare results, which has an --auto-build feature that seems similar to your autoclassify on opensuse
18:27:52 <bmwiedemann> so it tells you which variations matter?
18:27:53 <vagrantc> let's see if i can dig up an example
18:28:33 <vagrantc> bmwiedemann: it uses keywords instead of a bitmask, and also can run diffoscope on each resulting combination
18:29:02 <vagrantc> reprotest --auto-build --store-dir=$(mktemp -d) --min-cpus=10 --vary=-user_group,-domain_host,-fileordering auto -- null
18:29:41 <vagrantc> i disable some of the variations that require more privledges to set up, which means it cannot tests those variations, of course
18:30:02 <vagrantc> but for most things, i already know from tests.r-b.org weather i'm looking for one of those issues
18:30:50 <vagrantc> and if so, i can use sbuild to build as a different user, a virtual machine to change the hostname, etc.
18:31:31 <vagrantc> reprotest currently supports varying: +environment,  +build_path,  +kernel,  +aslr,  +num_cpus, +time, +user_group, +fileordering, +do- main_host, +home, +locales, +exec_path, +timezone, +umask
18:32:08 <bmwiedemann> so reprotest would return and say results did not differ in such a case, when you did not vary the relevant bits?
18:32:38 <vagrantc> with --auto-build it will tell you which of the variations matched a control build
18:33:10 <vagrantc> so it basically does a control build, and then does a build normalizing everything it can to see if it can be made reproducible
18:33:44 <vagrantc> e.g. comparing the default build, and then a normalized build against that to see if it is reproducible ... and then proceeds to test each individual variation
18:34:03 <vagrantc> it might also do a build with all the requested variations
18:34:49 <vagrantc> i use --store-dir to be able to look at all the various diffoscope outputs for each variation
18:34:58 <Foxboron> When looking at the CI and using reprotest, are you hitting repro issues you'd have if you only tried rebuilding the package according to the buildinfo file?
18:36:19 <vagrantc> Foxboron: haven't done much in the way of verification builds
18:36:50 <vagrantc> Foxboron: h01ger has done a bit more of that sort of thing with the attempt at rebuilders
18:37:14 <vagrantc> but mostly i'm just looking at and for issues and then submitting patches
18:37:52 <vagrantc> i.e. reproducing the issue locally, not necessarily the artifacts
18:38:04 <vagrantc> and then reproducing builds locally
18:39:24 <vagrantc> i'm not sure there's much else to my workflow, beyond the specifics of filing bugs ... i usually don't submit upstream, but to the debian bug tracker ... so i can have all my bugs in a consistent interface ... and hope that the debian maintainers for the package forward them upstream
18:40:31 <vagrantc> oh, i also look to see if the issue is fixed upstream, since very amazing people such as bmwiedemann *do* usually file upstream and sometimes i can just ask the debian package maintainer to include the upstream patch
18:41:04 <Foxboron> :D <3
18:41:49 <vagrantc> the rest is just basic debugging ... patch, rebuild, see the results, rinse, wash, repeat :)
18:41:56 <bmwiedemann> sometimes I also hit an issue and look at the Debian package for the relevant patch and test+upstream that
18:42:27 <vagrantc> bmwiedemann: i just saw yesterday a patch of mine you helpfully forwarded upstream :)
18:43:40 <vagrantc> anyone want to spell out their workflow from some other distro?
18:43:47 <bmwiedemann> only works well, if there is an openSUSE package
18:44:23 <vagrantc> oh, let me throw a few links in for the bot
18:44:38 <vagrantc> #link https://tests.reproducible-builds.org/debian/PACKAGE
18:45:02 <vagrantc> #link https://tracker.debian.org/reprotest
18:45:45 <vagrantc> #info reprotest --auto-build --store-dir=$(mktemp -d) --min-cpus=10 --vary=-user_group,-domain_host,-fileordering auto -- null
18:45:47 <Foxboron> I can quickly take the Arch stuff :)
18:45:51 <bmwiedemann> #link https://wiki.debian.org/ReproducibleBuilds/Howto
18:46:20 <vagrantc> bmwiedemann: really need to update that page ... a lot of the links should be moved to reproducible-builds.org/docs
18:46:27 <vagrantc> but yeah
18:46:40 <vagrantc> #topic debugging Archlinux packages
18:46:46 <vagrantc> Foxboron: you have the stage :)
18:47:48 <Foxboron> Yas!
18:48:08 * vagrantc hopes Archlinux is appropriately spelled/cased
18:48:13 <jelle> Our packages have no bugs
18:48:15 <Foxboron> So when I started to contribute back in 2017 things where sorta similar to vagrantc. Look at CI, figure out the difference, patch and see if it works.
18:48:19 <Foxboron> vagrantc: Arch Linux :D
18:48:23 <Foxboron> or archlinux, but never Archlinux
18:48:24 <vagrantc> doh.
18:48:33 <vagrantc> #topic debugging Arch Linux
18:48:43 <Foxboron> These days however we don't rely on the CI as much. We use the lovely rebuilderd kpcyrd wrote for us!
18:48:50 <Foxboron> #link https://reproducible.archlinux.org/
18:49:07 <Foxboron> This utilizes archlinux-repro which does verification builds towards distributed packages.
18:49:12 <Foxboron> #link https://github.com/archlinux/archlinux-repro/blob/master/docs/repro.8.txt
18:49:51 <Foxboron> this allows us to always recreate the results we see from rebuilderd, then we can build a new package, patch it, and see if we can reproduce. All using the same consistent tooling.
18:50:23 <Foxboron> The downside is that we don't have reprotest so any upstream reproducability issues isn't neccesarily found as when you use the CI system. But it gives us an advantage that we can always do 1:1 builds from what we see
18:50:50 <Foxboron> Now rebuilderd also has log output, so it's easier to have a starting point on the issues, and see what needs to be done.
18:50:52 <Foxboron> #link https://reproducible.archlinux.org/api/v0/builds/74/diffoscope
18:50:56 <Foxboron> #link https://reproducible.archlinux.org/api/v0/builds/74/log
18:51:17 <Foxboron> We also have the reprobuilds wiki page that sorta introduces all this, but it's not super well structured :/
18:51:18 <Foxboron> #link https://wiki.archlinux.org/index.php/Reproducible_Builds/Status
18:51:25 <Foxboron> noo,
18:51:27 <Foxboron> #link https://wiki.archlinux.org/index.php/Reproducible_Builds#Helping_out
18:51:28 <Foxboron> that one!
18:51:56 <Foxboron> So yeah, we have quite a nice workflow (i'd say) with repro and rebuilderd. But we also avoid the tricky issues as we only done one architecture when building
18:53:13 <Foxboron> A short link spam, but I hope that grouped up the stuff nicely :)
18:53:53 <bmwiedemann> so how do you find out what causes the variations in the output? e.g. date vs hostname vs readdir-order?
18:54:03 <vagrantc> i guess in theory you could add a mode to  ... repro ... to do reprotest-like things
18:54:28 <Foxboron> vagrantc: We sorta decided to keep it seperate. repro also does verification while if we need to have a difference between the build envs one should use reprotest
18:54:53 <Foxboron> bmwiedemann: So it depends a little bit. We have had issues where the repro/rebuilderd tooling is complicated enough to introduce issues.
18:55:27 <Foxboron> So sometimes you don't know if you are looking at a reproducability issue, or a tooling issue. This requires some "clever" hacks where we jump into the container and mess around :p it's not super straight forward.
18:55:47 <vagrantc> Foxboron: makes sense ... though i was curious for some of the systemd-nspawn integration if it would be feasible to re-use the code to spawn a reprotest environment
18:55:48 <Foxboron> As for figuring out the issues it's mostly the same. Git clone the project, try write a patch, then apply it on the package and run repro
18:56:03 <Foxboron> vagrantc: I think so, but I havne't really investigated
18:56:32 <bmwiedemann> for "write a patch" you need a clue what to look for, though.
18:57:22 <Foxboron> It's educated guessing :p Some stuff we have tried document. Like "PYC issues, look at PYTHONHASHSEED". But we haven't been good at that.
18:57:24 <bmwiedemann> I sometimes apply as many fixes as I can think of and then drop them one by one as long as it remains reproducible.
18:57:31 <vagrantc> sometimes my patches are just turning debugging up to the MAX :)
18:57:34 <Foxboron> haha
18:57:54 <Foxboron> I wanted to try include some container breakpoint into repro so you can go into the env, and build by hand to figure out any issues
18:58:02 <Foxboron> include git, then you can git commit stuff and get out patches :p
18:58:20 <Foxboron> But it would need some more work I think
18:58:41 <Foxboron> Say you can step into the build process and flick switches. let it continue and see what happened.
18:58:44 <bmwiedemann> I think, your workflow could profit from something like autoclassify
18:59:10 <Foxboron> bmwiedemann: as in the concept?
19:00:07 <vagrantc> basically performing a build while varying one thing at a time ... so you know weather it's a build path, timestamp, timezone, locale, etc.
19:00:11 <bmwiedemann> something that reduces certain variations to find out if they matter. e.g. setarch -R to disable ASLR and if it becomes reproducible, that narrows down the class of issues to look for
19:00:15 <jelle> Off topic has anyone checked out the new time namespace in Linux containers
19:00:34 <vagrantc> have not, sounds tasty
19:01:15 <vagrantc> so, we're a bit over time ... do we want to wrap up?
19:01:38 <Foxboron> bmwiedemann: Hmm, I like it. Since we use BUILDINFO to reproduce we a slightly more limited set of issues (i think)
19:01:38 <bmwiedemann> I'm using kvm -rtc base=$later
19:01:42 <vagrantc> i mean, i'm fine with continuing on
19:01:43 <Foxboron> (given the tooling works)
19:02:22 <jelle> Yeah we have, kernel, time, file order hmmm
19:03:26 <bmwiedemann> I'm fine continuing
19:03:38 <jelle> Anyone ever tried to classify based on diffoscope output
19:04:07 <bmwiedemann> jelle: I sometimes grep diffs for certain patterns, so I can fix multiple similar issues
19:04:09 <Foxboron> jelle: it struck me when bmwiedemann mentioned it :)
19:04:12 <vagrantc> Foxboron: thanks for sharing about Arch Linux :)
19:04:19 <vagrantc> #topic miscelaneous tips and tricks
19:04:21 <Foxboron> vagrantc: nps :D I hope it gave some insight
19:04:33 <vagrantc> bmwiedemann: or did you want to present?
19:05:07 <bmwiedemann> I think, my writeup (linked in the etherpad) is already very comprehensive
19:05:28 <vagrantc> jelle: i had actually thought about finding some university students who needed a project to work on to do classification based on diffoscope output
19:05:39 <bmwiedemann> though some shorthands like 'rbkt' for "rebuild with kvm+tweaks" are not so obvious
19:05:43 <jelle> vagrantc: I like it
19:05:59 <vagrantc> #link https://github.com/bmwiedemann/reproducibleopensuse/blob/devel/howtodebug
19:06:08 <Foxboron> vagrantc: Might be a nice GSoC project
19:06:10 <jelle> vagrantc: although if building is cheap that can be done as well :)
19:06:16 <jelle> Oh yeah!
19:06:31 <vagrantc> bmwiedemann: ah yes, if you can spell out some of those notations, that would be great :)
19:07:15 <vagrantc> Foxboron: indeed
19:08:36 <bmwiedemann> vagrantc: I vaguely remember we once had research of chinese researchers in our weekly reports that used heuristics to detect+fix rb-issues
19:09:07 <vagrantc> bmwiedemann: Debian's i386 builds do one build with a 686-pae kernel and one with an amd64 kernel, without using linux32 to set the personality ... finds some bugs that way ...
19:09:24 <vagrantc> bmwiedemann: it rings some bells
19:09:46 <vagrantc> faintly
19:10:04 <bmwiedemann> in #167
19:10:59 <bmwiedemann> so maybe we could try to apply that first
19:15:17 <bmwiedemann> vagrantc: the 'noarch' test I meant was different - If you build something that is not arch-specific like docs, and do it once on i386 and x86_64 - it should still produce the same docs
19:17:00 <vagrantc> bmwiedemann: ah, yes! we do build arch:all packages on all the architectures we test, amd64, i386, arm64, armhf ... but do not do any comparison of the results
19:17:12 <vagrantc> at least, not automatically ... i've done it manually here and there
19:17:25 <vagrantc> i would like to do such a test
19:18:21 <vagrantc> #link https://blog.acolyer.org/2018/06/22/automated-localization-for-unreproducible-builds/
19:18:44 <vagrantc> wow, why do we not have that integrated already?
19:18:46 <vagrantc> :)
19:22:18 <bmwiedemann> integration can be hard
19:23:54 <bmwiedemann> do we want to end for today?
19:25:03 <vagrantc> we can end the "meeting" and carry on like always :)
19:25:18 <vagrantc> oh, i had another quick point on guix i'll dump in the notes
19:25:24 <vagrantc> #topic debugging guix
19:25:35 <vagrantc> #info guix challenge PACKAGE
19:26:12 <vagrantc> #info guix build --keep-failed --no-grafts --check --rounds=2 PACKAGE
19:26:18 <vagrantc> #topic thanks all
19:26:22 <vagrantc> thanks everyone!
19:26:28 <vagrantc> #endmeeting