17:19:34 <vagrantc> #startmeeting
17:19:34 <MeetBot> Meeting started Mon Nov 30 17:19:34 2020 UTC.  The chair is vagrantc. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:19:34 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
17:19:55 <vagrantc> #topic office hours / ask me anything
17:20:22 <vagrantc> gchristensen: mind if we cut-and-paste your prior questions a bit for the bot?
17:20:29 <gchristensen> not at all
17:20:30 <jathan> vagrantc: Besides Debian and Guix have you done Reproducible Builds work for other Free Software communities or distributions?
17:21:27 <vagrantc> jathan: well, i've contributed patches to specific projects, both guix and debian are collections of other upstream projects, so we end up contributing to all of them
17:21:37 <vagrantc> or at least many of them :)
17:22:24 <vagrantc> jathan: but for me personally, i've mostly worked through debian; e.g. submitting bugs to bugs.debian.org and hoping the maintainers will forward upstream
17:22:49 <vagrantc> i know bmwiedemann tends to contribute directly upstream
17:23:34 <vagrantc> it would make for better logs to change the topic on each question ... but that takes a bit more driving
17:23:55 <vagrantc> still learning how this idea will work out :)
17:23:59 <jathan> vagrantc: Is this the right place in the RB Salsa repo https://salsa.debian.org/reproducible-builds/reproducible-website/-/tree/master/_docs in which that How To could be migrated?
17:25:09 <vagrantc> jathan: i'm not sure where we landed on how much distro-specific stuff should land on r-b.org or if it should stay with the respective distribution's documentation infrastructure
17:25:55 <vagrantc> should we continue to maintain wiki.debian.org separately and heavily reference r-b.org/docs where appropriate ...
17:26:36 <jathan> vagrantc: I see. Thanks for telling us about this :)
17:26:40 <vagrantc> jathan: but if we were to generalize it, i think that's where it should go ... and *maybE* add a distro-specific page there or something ... or reference the debian wiki, etc.
17:26:56 <vagrantc> at least cross-reference each other
17:27:07 <vagrantc> jathan: thanks for generating lots of questions :)
17:27:38 <jathan> vagrantc: Thanks to you for this great session!
17:29:25 <jathan> vagrantc: In your opinion this How To of bmwiedemann https://github.com/bmwiedemann/reproducibleopensuse/blob/devel/howtodebug could be adapted and applied to other projects besides OpenSUSE?
17:30:44 <vagrantc> jathan: a lot of that is pretty general, even for debugging issues unrelated to reproducible builds ... but yeah, maybe that would be an interesting starting point
17:30:59 <jathan> vagrantc: Ok.
17:31:54 <vagrantc> bmwiedemann might join us later, too, might be worth discussing then
17:32:06 <bmwiedemann> jathan: roughly 70% of my patches go upstream. the others to openSUSE
17:32:49 <vagrantc> jathan: also, doing an inventory of various distro-specific howtos and trying to generalize them into reproducible-builds.org/docs somewhere might be a good starter project :)
17:33:12 <jathan> bmwiedemann: Hi :) Thanks for sharing that answer.
17:33:31 <vagrantc> ah, maybe we should use #info for the question and #info for a summarized follow-up
17:33:55 <vagrantc> that way people other than the chair can add items
17:38:22 <jathan> vagrantc: It is a good idea! An apology to could not take any stronger commitment currently to say "Yes, I will do it" :D
17:39:42 <vagrantc> hah
17:40:45 <jathan> I really admire the skills and the work the core team have done since the beggining until now.
17:40:51 <jathan> By the way.
17:42:37 <jathan> I remember Profit Bricks was helping with servers for RB some years ago, but currently they are not part of the sponsors: https://reproducible-builds.org/projects/
17:43:10 <jathan> h01ger or vagrantc: Are they are not helping with that anymore?
17:43:20 <bmwiedemann> I think, the company renamed
17:46:34 <jathan> bmwiedemann: Trying to visit their website, now the URL redirect to IONOS, so thinking about you are sharing, maybe it has been bought by 1&1.
17:47:51 <jathan> And IONOS is a non fiscal sponsoer of RB. Now I understand.
17:48:44 <vagrantc> yeah, i think that is what happened
17:49:15 <jathan> bmwiedemann and vagrantc: Thanks.
17:50:43 <h01ger> jathan: profitbricks has renamed itself to ionos and has been sponsoring us since 2012 (well, r-b only started to use those ressources in 2014, so us=jenkins.debian.net here)
17:50:59 <vagrantc> jathan: from jenkins git repository: git log --patch --grep=profitbricks
17:51:05 * h01ger is only partly here so answers may be delayed a bit
17:51:18 <jathan> vagrantc: How do you manage to work in RB and your other activities as job or is RB your main job?
17:51:54 <vagrantc> jathan: i've mostly worked part-time for RB and that leaves time for other things, at least in theory :)
17:52:34 <jathan> h01ger: Hi! :) Thanks a lot for sharing this information. Yes, I was thinking about the jenkins hosting with them :)
17:52:36 <vagrantc> jathan: so mostly manage by limiting my commitments
17:53:44 <vagrantc> which sometimes is a bit hard, as there's so much to do
17:53:52 <jathan> vagrantc: Amazing, I was thinking maybe it was your core job :)
17:54:55 <vagrantc> jathan: probably just the context you know me in
17:56:08 <jathan> vagrantc: I see :)
17:57:55 <vagrantc> jathan: i guess another thing that allows me to do part-time stuff is i use one of the big donated servers to run reproducible builds testing in the background, and then go do other things which might not be related to reproducible builds, and then come back to check on the results
17:58:21 <vagrantc> at least for build jobs that take a long time, that works
17:58:32 <vagrantc> no sense staring at a progress bar :)
18:00:00 <jathan> vagrantc: Are those RB testings for Debian, Guix or in general like the called upstream?
18:00:21 <vagrantc> jathan: mostly for debian
18:00:56 <vagrantc> although, now that i maintain guix *in* debian, sometimes it's not an either-or question :)
18:01:22 <vagrantc> e.g. have been looking at reproducibility issues in guix itself since i build debian packages of it now
18:02:01 <vagrantc> for that one, i'm building guix on debian, finding reproducibility issues, and submitting upstream to guix :)
18:02:40 <vagrantc> since guix does more normalization, there are some things they don't catch with their own tests
18:03:10 <vagrantc> perfect example of Quality Assurance type build vs. normalized build to improve reproducibility
18:03:14 <jathan> vagrantc: Thinking about the rebuilderd for Arch h01ger and kpcyrd shared with me in the mailing list. Does Guix is using currently rebuilderd or it is missing it too as Arch?
18:03:33 <h01ger> rebuilderd is arch only atm
18:03:54 <jathan> h01ger: Thanks brother :)
18:04:03 <vagrantc> guix uses guix ... it has some features built-in, such as "guix challenge PACKAGE" and "guix build --rounds=N PACKAGE"
18:04:34 <vagrantc> it's not a coincidence that i work on reproducible builds and got interested in guix
18:05:26 <vagrantc> working on debian, where we've been working on integrating reproducibility into a project that is decades old ... vs. guix as a project that was thinking about reproducibility almost from the start
18:05:35 <jathan> vagrantc: Very interesting the cycle of maintaining Guix in Debian and the relation to submit upstream to Guix :)
18:06:36 <h01ger> btw, jenkins is currently not running any jobs as its in shutdown mode to install plugin updates and shutdown takes 24h because of long running fdroid jobs
18:06:52 <vagrantc> wow. TIL. :)
18:07:49 <bmwiedemann> vagrantc: I was thinking to package Mes and maybe parts of guix in openSUSE. Would be nice to hear hints from you for that.
18:08:38 <jathan> vagrantc: Yeah, it is wonderful you started to work in RB in Debian before Guix and then that advantage and experience, let you have in mind the RB needs in Guix at the beggining.
18:08:40 <vagrantc> bmwiedemann: i think guix is already in opensuse
18:09:06 <vagrantc> bmwiedemann: i think jonsger packaged it ...
18:09:24 <vagrantc> bmwiedemann: in fact, i think i even reference an opensuse bug regarding reproducibility of guix :)
18:09:41 <jathan> vagrantc: Thank you very much for your time for this session! As well h01ger and bmwiedemann :)
18:09:59 <vagrantc> jathan: oh, guix has been doing reproducible builds before i ever heard about it
18:10:33 <jathan> vagrantc: Oh! Really? I thought Guix was created after 2014 ha.
18:10:47 <vagrantc> so, in theory the Q&A goes till 20UTC ... of course, after an hour of questions ... we'll see :)
18:11:01 <bmwiedemann> indeed, guix is available in openSUSE already :-)
18:11:26 <vagrantc> jathan: guix started in 2013, and it was conceptually based on nix from earlier than that, talking about deterministic builds
18:11:48 <vagrantc> bmwiedemann: the bug was actually reported by ... someone you know very close :)
18:12:41 <vagrantc> bmwiedemann: https://bugzilla.opensuse.org/show_bug.cgi?id=1170378
18:13:22 <vagrantc> i guess that was on guile...
18:14:16 <bmwiedemann> guile+guix
18:14:37 <bmwiedemann> it is a guile toolchain issue though
18:14:45 <jathan> vagrantc: Thanks for pointing that about Guix :)
18:15:16 <vagrantc> jathan: according to wikipedia, nixos was started in 2003: https://en.wikipedia.org/wiki/NixOS and pretty sure it mentions build determinism in the original academic papers
18:15:53 <vagrantc> bmwiedemann: as far as packaging mes ... does opensuse have an i386/x86 port? or cross-compilers for it?
18:16:10 <vagrantc> bmwiedemann: i think that was a stumbling block when jelle tried to package mes for archlinux
18:16:32 <bmwiedemann> vagrantc: yes, we have a full i586 distribution of openSUSE-Tumbleweed plus the 32bit subpackages in the x86_64
18:16:33 <vagrantc> you can build a version that's just built with gcc, but not mes building mes
18:16:58 <vagrantc> bmwiedemann: but of course, happy to answer any specific questions you have about it :)
18:17:23 <vagrantc> bmwiedemann: there's also #bootstrappable on irc.freenode.net which might rope in other people who know about it
18:18:05 <jathan> vagrantc: I didn't read about it before. It is good to know now about the build determinism of NixOS.
18:18:55 <bmwiedemann> I think, NixOS and Guix are conceptually very close. They just use different languages for their build descriptions.
18:18:56 <h01ger> btw, i'm thinking about dropping i386 from the debian tests (and removing all history except maybe graphs and a few datapoints). for two reasons: a.) i386 is dead/slowly dying, hardly anyone still uses it b.) but it uses 25% of the storage for us. c.) we want/need more storage for bookworm and whatever comes after d.) our i386 nodes frequently need to be kicked back into life with a reboot e.)
18:18:56 <h01ger> we will keep being able to test 32 bit variations on armhf. what do you think?
18:19:14 <vagrantc> jathan: also, nix is also packaged in debian
18:19:19 <h01ger> just curious right now, before doing so i will run this through our mailinglist (and debian-devel@l.d.o too)
18:19:37 <h01ger> s#two reasons#these reasons# :)
18:19:46 <vagrantc> h01ger: and here i always thought armhf would be the first to go
18:19:48 <vagrantc> :)
18:19:55 <h01ger> hah(a)
18:20:47 <vagrantc> h01ger: they might need to be kicked less if we lowered the available ram ... i know that means more builds would run out of memory, but i think that is a huge part of the problem ...
18:20:58 <vagrantc> h01ger: though that obviously doesn't address the other points
18:21:57 <vagrantc> the bug where i386 doesn't really handle PAE very well due to bugs in the kernel not likely to ever get fixed
18:22:14 <vagrantc> mostyly due to "a"
18:22:20 <bmwiedemann> vagrantc: I remember problems with linking firefox on i586, because 3GB addressable RAM was not enough.
18:22:56 <bmwiedemann> and as software becomes more bloated and linking (with LTO) becomes more expensive, this will only get worse
18:23:03 <vagrantc> indeed
18:23:57 <bmwiedemann> my last 32-bit only machine was a eeepc from ~2008 ; then next one from 2010 already had an atom that could run 64bit ops in 4GB RAM
18:24:17 <h01ger> vagrantc: right, i forgot f.) doing so will free ressources for other things
18:24:18 <vagrantc> h01ger: i have sometimes found incredibly useful to see bugs that consistently fail on 32-bit or consistantly fail on arm*, so that would remove one data point for such observations
18:24:39 <h01ger> vagrantc: right
18:25:38 <vagrantc> h01ger: regarding storage ... would it make sense to solicit more storage capacity somehow? i suspect we could find donations ... but then there's a maintenance cost and having to integrate some other service...
18:26:16 <h01ger> its storage in the ionos cloud..
18:27:59 <vagrantc> there's a balance to be struck between complexity and scalability, but sometimes wondered if splitting out some of the infrastructure by architecture rather than having a single jenkins running all the architectures ... e.g. and then use the stuff that communicates from other distros or something ... but jenkins is a beast to manage, managing more of them sounds terrifying :)
18:28:02 <h01ger> vagrantc: so having i386 logs has sometimes been useful for you. i guess that could be kept by keeping i386/unstable instead of i386/all, right? (though i'm not sure if that wouldnt make things more complicated in turn, which i would not want)
18:29:05 <vagrantc> h01ger: and maybe skipping experimental too, though that's a drop in the bucket ... not sure how much more complex that would be
18:29:19 <vagrantc> h01ger: or just building $testing instead of unstable
18:29:24 <h01ger> its either keeping only unstable or skipping them all
18:29:35 <vagrantc> h01ger: maybe that's enough to sum up and carry to the list?
18:29:35 <h01ger> or keeping only testing or skipping them all
18:29:39 <h01ger> sure
18:30:14 <vagrantc> also the armhf tests don't systematically test time variation, the i386 do?
18:30:38 <vagrantc> we'd also loose a language-specific variation, since each arch tests C and $LANG
18:31:54 <h01ger> that language specific variation is neclectable
18:31:59 <h01ger> i'd say
18:32:04 <vagrantc> ish :)
18:32:16 <h01ger> date variation is tested on amd64 and arm64, so meh
18:33:38 <h01ger> vagrantc: we are testing variations with 4 roman languages and C, currentl. we dont test right to left languages or any other glyph than ascii. so yes, i'm fairly confident to say that the language variation on i386 (which is one of those 4 roman languages) is neglectable
18:33:43 <vagrantc> i'm going to grab some food, but will be back shortly if anyone wants to shout out more Q&A
18:33:54 <h01ger> vagrantc: bon appetit!
18:34:17 <h01ger> vagrantc: to ask something myself: when will the next AMA session be?
18:36:48 <bmwiedemann> I once found swiss locale being special because of dot vs comma
18:37:03 * bmwiedemann is off
18:38:54 <vagrantc> h01ger: hmmm... was figuring i'd ponder that a bit after this one was "done". :)
18:39:37 <vagrantc> and there was that one bug i found in u-boot specifically with it_IT locale :)
18:42:46 <h01ger> vagrantc: i just want to move todays event further ahead in my calendar (instead of deleting it now and recreating it tomorrow) - but so be it :)
18:43:35 <h01ger> bmwiedemann: vagrantc: sure, there are interesting locales. but we can vary those too, we dont need to keep an arch for that :)
18:43:52 <vagrantc> h01ger: i'm thinking no more than once a month, but not sure ... it's also the kind of thing i can do other RB work if there are no/few questions at the same time slot
18:44:48 <vagrantc> and also, might need to just accept some of these where nobody shows up, but keep doing them and getting the word out ...
18:45:19 <vagrantc> also wondering the wisdom of a three hour session :)
18:45:53 <vagrantc> but again, as long as i can keep it open while working on other things ... it's just an open invitation
18:58:44 <h01ger> vagrantc: i think once a month is great
19:01:55 <vagrantc> a bit tricky to schedule once-a-month with a bi-weekly general rb meeting ... but we'll see
19:02:14 <h01ger> different day
19:02:18 <vagrantc> true
19:07:03 <vagrantc> hmmm... thinking out my own personal availability ... probably January 7th, 18:00-19:00 UTC
19:07:20 <vagrantc> maybe i'll just call it so we can get it in the next monthly report
19:08:10 <h01ger> maybe make it 2h?
19:08:27 <h01ger> besides that, i like to define a date now/soon
19:09:46 <vagrantc> 17UTC is admittedly pretty early for me ... but 20UTC end time is pretty late for some people ... but i'd be most inclined to 18-20UTC
19:10:30 <vagrantc> so i'd go with 2021-01-07 18:00-20:00 UTC ... after having mostly consulted myself :)
19:10:54 <h01ger> as long as you run this, you should listen to yourself a lot and weight yourself heavily too :)
19:11:36 <vagrantc> indeed
19:12:15 <h01ger> btw: ssh: connect to host jtx1c-armhf-rb.debian.net port 2254: No route to host
19:12:27 * vagrantc sighs
19:12:39 <vagrantc> just fixed it last night ... but who knows what happened while i slept
19:13:33 <vagrantc> ugh, looks likely to be more disk issues
19:13:46 <h01ger> ugh
19:16:27 <h01ger> vagrantc: btw, please keep the '# need(s) investigation\n# none atm" lines or line in jenkins-home/offline nodes, usually easiest done by reverting the commit which marked a host offline
19:16:34 <h01ger> (super minor detail, but still)
19:17:58 <vagrantc> h01ger: well, once i've done the investigation, i've been recording the reason ... ?
19:18:19 <vagrantc> or removed it
19:23:28 <h01ger> and then tomorrow the next node needs investigation and i'm gonna re-add that phrase. thats why i like to keep it :)
19:25:04 <kpcyrd> vagrantc: does guix have a package database or is guix cloning and sourcing the code in ./gnu/packages/ in https://git.savannah.gnu.org/git/guix.git/?
19:25:55 <kpcyrd> (guix the package manager)
19:26:47 <vagrantc> kpcyrd: there's a local sqlite database in /var/guix/ ... and there's metadata created by the guix build service project that chris baines has been working on
19:26:51 * vagrantc rummages around for links
19:27:20 <vagrantc> h01ger: ah, you mean keep the phrase in there regardless of weather there are any machines affected... got it!
19:28:51 <h01ger> vagrantc: yes. its either "# need investigation\n$host1\n$host2" or "# need investigation\n# none atm"
19:30:15 <vagrantc> kpcyrd: https://data.guix.gnu.org/ might be the sort of thing you're looking for?
19:31:23 <vagrantc> kpcyrd: also, guix's rough equivalent to "apt get update" is "guix pull" which downloads the latest git (or a specific git you want) and essentially compiles the available package data
19:32:06 <kpcyrd> vagrantc: ah I see, that's kind of what I assumed :)
19:34:31 <vagrantc> in theory, if you've only changed a small number of packages it doesn't have to recompile much ... but due to the nature of inter-dependencies between various package sets ... it often requires recompiling a lot
19:35:35 * vagrantc waves hands a bit
19:36:06 <kpcyrd> vagrantc: `guix challenge PACKAGE` only takes the package name as argument, right? and the version is taken from the database? can I provide a path to the expected package instead of automatically downloading it from a mirror?
19:39:02 <vagrantc> kpcyrd: the version and hash of inputs is what "guix pull" compiles ...
19:40:47 <vagrantc> kpcyrd: so the version of "guix challenge PACKAGE" gives you the derivation for PACKAGE which would typically be something like /gnu/store/HASHOFINPUTS-PACKAGE-VERSION.drv ... which can be used to download and/or build /gnu/store/HASHOFOTHERINPUTS-PACKAGE-VERSION ... and then guix challenge checks if that matches
19:41:22 <jelle> vagrantc: still need to get back at my mess with mes
19:41:31 <vagrantc> jelle: :)
19:42:45 <vagrantc> i think guix challenge can even run diff or diffoscope for you to compare
19:43:41 <vagrantc> cbaines: oh, you're here too :)
19:43:50 <cbaines> yah :)
19:45:07 <cbaines> kpcyrd, are you're questions related to rebuilderd and Guix?
19:45:13 <vagrantc> kpcyrd: guix can also have multiple concurrent avilable versions ... in that case i *think* it can also take PACKAGE@X.Y to specify the version
19:45:33 <vagrantc> kpcyrd: though sometimes in guix i'm unsure of weather the @VERSION syntax works
20:00:52 <vagrantc> ok, i'm going to spam some re-post to get it into the meetbot logs and then close the meeting
20:01:09 <vagrantc> < gchristensen> what do you think about reproducible builds in the archival, long-term-does-it-still-build sense?
20:01:12 <vagrantc> < vagrantc> you mean reproducing an old build of software, or reproducing a build of software today in the
20:01:15 <vagrantc> future?
20:01:17 <vagrantc> < gchristensen> today, in the future
20:01:20 <vagrantc> < vagrantc> i think it is a great application of reproducible builds :)
20:01:23 <vagrantc> < gchristensen> are reproducible builds enough?
20:01:25 <vagrantc> < vagrantc> well, you of course need to track the information necessary to reproduce a build ... e.g. the
20:01:28 <vagrantc> .buildinfo file (or guix commit hash, in the case of guix)
20:01:31 <vagrantc> < vagrantc> but the current assumption with reproducible builds is that your toolchain remains the same, so
20:01:34 <vagrantc> you'd essentially need to either archive all your build-dependencies or have each of those be
20:01:37 <vagrantc> reproducible from source as well
20:01:40 <vagrantc> < vagrantc> gchristensen: is that touching on what you were getting at? or elaborate on "enough" :)
20:01:43 <vagrantc> < vagrantc> but rebuilding all your software from source starts to edge outside of reproducible builds and into
20:01:46 <vagrantc> bootstrappable builds
20:01:48 <vagrantc> < vagrantc> which is also good :)
20:01:51 <vagrantc> < gchristensen> well I guess I was hoping to just start some discussion in general :). I'm thinking about the
20:01:54 <vagrantc> times I tried to build a very old emacs from nixpkgs and had trouble, but I don't remember what
20:01:57 <vagrantc> exactly. ... but related topics are things like source archiving, glibc locale issues, things
20:02:00 <vagrantc> that change over time: timezones, certificates, locales ...
20:02:03 <vagrantc> < gchristensen> maybe emacs is unfair given unexec
20:02:05 <vagrantc> < vagrantc> gchristensen: for the debian infrastructure, we intentionally vary some things that things like nix
20:02:08 <vagrantc> will normalize in the build infrastructure to try and catch issues like that
20:02:11 <vagrantc> < gchristensen> good point!
20:02:14 <vagrantc> < vagrantc> so there are largely two strategies to that ... normalize nix/guix and some tools i've heard of from
20:02:17 <vagrantc> some academic works attempt ... or to vary to try and detect as many issues as possible
20:02:20 <vagrantc> < gchristensen> it would be interesting to try introducing more variability in builds
20:02:23 <vagrantc> < gchristensen> in nix builds*
20:02:23 <vagrantc> #endmeeting