17:58:18 <spwhitton> #startmeeting
17:58:18 <MeetBot> Meeting started Wed Oct 21 17:58:18 2020 UTC.  The chair is spwhitton. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:58:18 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
17:58:29 <spwhitton> #topic Roll call
17:58:41 * bremner David Bremner sortof here
17:58:41 <smcv> Simon McVittie
17:58:45 * gwolf Gunnar Wolf
17:58:45 <spwhitton> Sean Whitton
17:59:49 <fil> Philip Hands
18:00:52 <ehashman> Elana Hashman
18:02:06 <spwhitton> #topic #971515 - kubernetes: excessive vendoring (private libraries)
18:02:30 <spwhitton> I tried to write a framing post yesterday.  Dmitry does not really agree.
18:02:54 <smcv> thanks for responding to that bug, spwhitton
18:03:06 <spwhitton> What thoughts do people have?  For my part I would like to know whether I basically got the dispute right or whether I missed things.
18:03:19 <ehashman> as I previously mentioned, I have Strong Opinions on this one. I've been holding off on posting until we met
18:03:33 <spwhitton> ehashman: now seems a good time to share them :)
18:04:37 <ehashman> I think this bug was sort of inevitable. Debian policy cruft is clashing with how people actually build and distribute software these days. we have an appeal to override a maintainer on the basis of "policy" and the "Debian way being superior" without any real technical examination of the merits of "the Debian way"
18:04:44 <gwolf> I have to confess, I have not had time to read the answers to spwhitton :-( (but I did read your pessage)
18:05:02 <gwolf> s/pessage/message/ of course
18:05:33 <spwhitton> ehashman: it seems to me that we have to think about this sort of stuff package-by-package.
18:05:47 <smcv> what ehashman said there is something that struck me too, while reading the bug mails. it's talking about de-vendoring being part of Debian's identity
18:05:56 <smcv> but if that's the case, then IMO it shouldn't be
18:05:56 <spwhitton> It might be reasonable to vendor like mad for something written in Go but not for something written in Haskell, say
18:06:17 <smcv> our identity should be about shipping high-quality packages, whatever that means
18:06:19 <ehashman> Debian policy is specifically built around the distribution and execution of dynamically linked C/C++ applications and libraries. distros in general were. but modern software above that base does not necessarily operate under the same assumptions and it is silly to apply policy that was designed for applications that are dynamically linked against programs that are statically linked and are ...
18:06:19 <ehashman> ... impossible to dynamically link
18:06:28 <ehashman> wow thanks splitlong, lol
18:06:36 <gwolf> ehashman: You mention it right, and I like you not fearing saying that "we think our way is superior". I do think it _keeps_ being superior for most cases... but there are probably many cases where, yes, the Debian way does not fly anymore in 2020
18:06:46 <smcv> if de-vendoring is what gives us the highest-quality packages, then we should do it
18:07:00 <spwhitton> smcv: right.  and sometimes it does, and sometimes it doesn't.
18:07:02 <smcv> but if vendoring dependencies is what gives us the highest-quality packages, then we should do that
18:07:23 <ehashman> agree. and if not de-vendoring gives us the highest qualify packages, then we should do that. in this case, I think not de-vendoring results in a higher quality package for significantly less work
18:08:25 <smcv> and yes, this is coming back to what we talked about with (not) splitting JS stuff
18:08:47 <smcv> our packaging policy is designed for medium-sized packages
18:08:50 <spwhitton> it seems we're being asked to decide whether vendoring is what gives us the higher quality package in this case.  Elana has a settled view on that because she's involved in k8s.  how can the rest of us come to a settled view on it?
18:08:52 <ehashman> the main justifications for "de-vendoring" I think of as follows: ensuring each dependency is properly vetted for DFSG and ensuring easier security patching for dynamically linked executables
18:08:57 <gwolf> ehashman: Yes... and no. Often, keeping vendored libraries in will result in very important defects being hidden and not considered
18:09:13 <ehashman> in this case, the former (DFSG) is a) moot and b) not ctte's call, that's an FTP Team thing
18:10:13 <ehashman> and devendoring makes security patching _harder_ when applications are statically linked. devendoring also operates on the assumption that it is good and desirable to only have a single version of a library or dependency on an entire system. modern software development practices staunchly disagree
18:10:16 <smcv> DFSG considerations are relevant inasmuch as it takes a long time to check vendored libraries for Freeness
18:10:19 <gwolf> ehashman: It's not our call to judge whether _a specific_ package is DFSG-free, but it _is_ our call to emit an opinion on whether a given practice is considered acceptable
18:10:46 <spwhitton> it's true that if we had a lot more vendored packages, current NEW practices would buckle horrendously
18:11:11 <smcv> every time I import a new version of mozjs or libsoup that's several hours I'll never get back
18:11:11 <ehashman> gwolf: I mean, in my personal opinion, if Debian doesn't consider Kubernetes as released from the official repos DFSG-free, our DFSG-checking process is wrong :)
18:11:17 <smcv> and presumably the same for the ftp team
18:12:03 <spwhitton> ehashman: that's fair enough but we have to decide what's reasonable given the ftp team's current views on things.
18:12:05 <ehashman> I don't want to dump on the FTP Team, and I do want to keep the scope of the discusion limited to the k8s package, where lawyers from companies over have already vetted the deps tree
18:12:06 <fil> I worry that packages where vendoring seems like the right way to go are effectively packages where security support becomes "install the latest" in which case they don't blong in stable, because they'll be starting to rot before anyone gets to use them.
18:12:51 <ehashman> fil: that is not the case with k8s though. they backport security patches to the latest 3 major release branches. but if we devendor it makes it significantly harder to take those patches
18:13:02 <gwolf> ehashman: Thing is, AIUI, we are not discussing about K8S, but about a bundling practice to which K8S subscribes
18:13:03 <smcv> fil: not necessarily - e.g. libsoup (which has a bunch of vendored rust) has stable-branches
18:13:26 <smcv> sorry, not libsoup
18:13:27 <smcv> librsvg
18:13:44 <gwolf> ...My take is that we have to be very careful when deciding what is acceptable as vendorizing and what not. But we cannot be _too_ subjective here, otherwise the FTP-master team will hate us (and everybody will hate them)
18:14:21 <spwhitton> gwolf: can't we qualify our response to this bug by saying we're expressing a view on what's appropriate for this package and do not intend to settle bigger vendoring questions for Debian?
18:14:35 <smcv> I think we need to make sure our self-imposed rules don't result in "Debian cannot ship a useful distro any more"
18:14:48 <ehashman> smcv: +1000
18:15:02 <gwolf> spwhitton: Yes, but no matter what we do, we will be setting some precedents. So, we have to keep the bigger picture in mind.
18:15:19 <ehashman> gwolf: maybe it would be useful to explicitly set the scope of the discussion. k8s is quite typical for a golang application so I think any decision we make about k8s probably affects that whole ecosystem, but does not necessarily translate to e.g. JS packages
18:15:37 <gwolf> I mean, I will probably agree that K8S has to be included - but many other vendored projects will have to be rejected *because of non-careful vendoring*
18:15:40 <spwhitton> ehashman: is the scale of vendoring in k8s comparable to other go packages?
18:15:45 <ehashman> spwhitton: yes
18:15:50 <smcv> if we set policies that rule out shipping firefox, or chromium, or GNOME, then we no longer have a useful distro, IMO
18:15:52 <spwhitton> okay.  I was under the mistaken impression it was larger.
18:16:01 <ehashman> spwhitton: oh, well, for applications of its size.
18:16:11 <ehashman> Kubernetes is quite a large application/codebase
18:16:15 <spwhitton> ehashman: are there lots of those we're likely to want in Debian?
18:16:44 <smcv> (firefox bundles a lot of rust, GNOME depends on some packages with bundled rust too; rust is not the same as go but it has some of the same properties from a distro PoV)
18:17:13 <ehashman> off the top of my head? Kubernetes, Docker, InfluxDB, Istio, Hugo, Terraform, CockroachDB
18:17:25 <spwhitton> okay, that's a fair few
18:18:03 <ehashman> and, disappointingly to me, most of these applications have basically given up on OS packaging, they either provide their own debs via a PPA or generally say "do not install using a distro package manager ever"
18:18:25 <spwhitton> we've been discussing the needs of users so far.  in my mail I wrote about the needs of people trying to work with the source package.  does anyone have thoughts on that?
18:18:44 <ehashman> and since Go binaries are statically linked, they are trivial to distribute outside of a distribution
18:19:22 <gwolf> ehashman: For packages such as kubernetes and docker, that quite surprises me. How do they work with the "enterprisey" distribtions? I know RedHatters push K8S quite a bit... and I'd be surprised to learn it's not via a in-distro package!
18:19:27 <ehashman> spwhitton: I can tell you that it is about 10x easier to build a self-contained source package than it is to try to juggle 20+ dependencies in Debian :)
18:19:55 <ehashman> (source: am maintainer of a package with 20+ dependencies in Debian and ugh. frankly the only way it works is that nothing else relies on said dependencies so there's no version creep/competition.)
18:19:58 <spwhitton> ehashman: right.  but other stuff -- trying to update particular dependencies etc?
18:20:16 <ehashman> why would you do that?
18:20:36 <spwhitton> I mean it seems like something that free software should enable.
18:20:38 <gwolf> ehashman: if a given libfoo moves from the compatibility version you are using for your code...?
18:21:00 <spwhitton> off the top of my head, to get a version which has improved performance, if you're identified a bottleneck there.
18:21:02 <ehashman> spwhitton: I mean, sure, and you can do it just the same as you would patch the source code of the application itself.
18:21:24 <gwolf> I mean, my main issue with vendored dependencies is that _many_authors_ (I don't mean K8S in particular) decide to keep running an ancient dependency that is buggy and unmaintained
18:21:29 <ehashman> gwolf: that wouldn't happen in the vendored dep case though. you already have the version of the dependency you need and it's not going to change out from under you. this is why people do this
18:21:34 <spwhitton> playing devil's advocate: suppose Debian's go team decides that a particular library should be updated to improve performance, say
18:21:44 <gwolf> ...and that's a liability that distribution-wide libraries often solve better...
18:21:57 <spwhitton> if the TC has made it okay to have lots of copies of it, are we making their lives a lot worse?
18:22:17 <ehashman> spwhitton: what if a change in that version of the library in say Kubernetes results in a breaking behavioural regression?
18:22:28 <smcv> does the go ecosystem make it possible to de-vendor individual libraries that you particularly care about?
18:22:54 <spwhitton> ehashman: I mean sure, vendoring helps in that case.  I'm just considering the usecases we might be making worse.
18:23:04 <gwolf> ehashman: OTOH, what happens if _not_ chanigng said version makes K8S vulnerable to code injection due to a vulnerable library (that's long-fixed in the distribution)
18:23:12 <smcv> every time I've looked at de-vendoring some of the rust in librsvg, that has been what stopped me (apart from the whole not knowing rust thing)
18:23:13 <ehashman> smcv: not really. the reason that Go does vendoring is because it doesn't have a particularly functional dependency management system
18:23:25 <smcv> you can have vendored rust
18:23:33 <smcv> or you can have de-vendored distro rust libs
18:23:55 <smcv> what you can't have is, afaics, is "de-vendor just these three" or "vendor just these three"
18:24:08 <ehashman> gwolf: why would we not report the issue upstream?
18:24:25 <spwhitton> ehashman: we would, but we ought to be able to fix it ourselves too, surely?
18:24:33 <smcv> if that was straightforward then vendoring would be a less drastic all-or-nothing decision
18:24:57 <ehashman> it's possible the upstream developers don't care about security, but this is one of those cases where I think upstream is much more likely to be responsive and resourced than distros
18:25:30 <fil> ISTR it being claimed that most people want to be running the exact version that has passed all the tests run by upstream when it comes to K8s ... in which case chopping the code around for whatever reason seems unlikely to make anyone happy -- is that actually true?
18:25:44 <spwhitton> ehashman: right.  but we ought to be empowered to make our own security decisions.
18:25:50 <ehashman> smcv: I'm sure there's some means of being able to pick up source elsewhere for a build, via e.g. GOPATH hacks
18:26:09 <ehashman> spwhitton: sure, nothing is stopping us from patching the vendored libraries.
18:26:30 <smcv> I think it's a requirement that we *have the option* of patching the vendored libraries
18:26:31 <gwolf> ehashman: But, again, judging on whether K8S' authors are a competent bunch, while foobar's authors are not likely to respond in a timely way to our bugs... will put a lot of unneeded stress on ftpmasters
18:26:41 <smcv> which we do, so that's fine
18:26:42 <ehashman> fil: I think yes. unless Debian wants to start running its own suite of Kubernetes compliance tests
18:27:12 <smcv> but it's entirely possible that there are not very many practical situations in which we would *want to* take that option
18:27:34 <ehashman> gwolf: yes, I think that is a big part of the tension. sometimes, you have an upstream where the authors are less resourced and responsive than the downstream. this case is almost certainly the opposite
18:27:35 <smcv> analogous: we have the option of arbitrarily patching the Linux kernel
18:27:51 <fil> ehashman: that makes it seem like what we actually need is a decent way of letting our users install the upstream binaries, rather than trying to force those packages into Debian as a special case
18:27:58 <smcv> but propose a non-essential patch that has not gone upstream and see how happy it makes our kernel maintainers :-)
18:28:07 <spwhitton> fil: this is the sort of conclusion I find myself coming to as well.
18:28:24 <ehashman> fil: my question for you would then be, what applications are in scope for inclusion in a distribution?
18:29:09 <ehashman> is the future of Debian "here's a glibc platform base" ala manylinux scope and then everything else is someone else's problem? do we become some sort of nix-ish thing?
18:29:18 <smcv> this seems like the same line of reasoning that leads to endlessOS, which ships a GNOME-based desktop with approximately no apps in the OS layer
18:29:30 <smcv> want a user-facing app? flatpak is -> over there
18:29:43 <spwhitton> smcv: well, we could provide a flatpak repo with various DFSG and trust properties, as you explained in a debconf talk once
18:29:44 <gwolf> ...That would break some of the best of Debian's properties
18:29:54 <smcv> I think there's benefits to that, and also costs
18:29:59 <gwolf> although, yes, that's subjective - And I am a subject that grew up quite a bit ago...
18:30:12 <ehashman> gwolf: I agree. which is why I think the scope of applications included in Debian needs to be wider than just a common base with flatpaks or similar
18:30:44 <spwhitton> We've been discussing this for a while now.  Since we have other topics on the agenga, I think we should think more concretely about next steps for the bug.  ISTM that we need security team input.  And then it seems like no-one thinks we should override the maintainer in this case, we should just be careful how we word our response?
18:30:50 <ehashman> it's why I maintain Leiningen. being able to "apt install leiningen" significantly reduces a barrier to entry
18:31:04 <ehashman> even though you could always install it via the upstream means
18:31:28 <fil> ehashman: well, if the reality is that the users are only really going to be happy with that exact version, I suppose what I'm asking is: what is point of going to a lot of effort trying to give them something else?
18:31:52 <smcv> fil: I think there's maybe a useful distinction to be drawn here between upstream's binaries and upstream's source code
18:32:31 <smcv> a lot of maintainers consider delta vs upstream in the Debian packaging to be a bug
18:32:48 <smcv> maybe a less severe bug than what it's there to fix, so we do it anyway
18:33:24 <ehashman> smcv: myself included. delta should be minimized as much as possible because users expect when they install X they get X, not "X with significant modifications", and upstreams tend to resent distros packaging "X with significant modifications" when distro bugs get reported to them directly
18:33:47 <smcv> but the barrier to entry on applying hotfixes to a package you already compile from source is much much lower than if it was "first, figure out how to compile it..."
18:34:26 <ehashman> fil: I think the main benefits Debian provides by taking $upstream_app source and building it are independent builds + system integration + handy stuff like init system integration, prebaked configs, etc.
18:34:27 <gwolf> I think there's broad consensus that modifications to upstream should be kept at a minimum... If anything, patching for sane configuration settings or for better in-distro integration
18:34:58 <gwolf> We used to have much larger deltas years ago, and I think it was a slow but clear realization that... we should keep them to a minimum
18:35:43 <smcv> see also the meme of "I found a (bug|missing feature|necessary piece of infra), I fixed it in Debian, job done!"
18:36:05 <ehashman> basically, Debian policy has a lot of value, we just need to ensure there's always a good technical justification for that value :) and policy that dictates how applications should run, where configs live, etc. etc. are all still valuable to something like k8s even if the packaging policy around deps and vendoring does not really apply
18:36:08 <smcv> which tends to lead to the Debian-specific solution being replaced by a cross-distro solution years later
18:36:17 <gwolf> of course. Upstream authors should not need to fish for their patches in our repository
18:36:22 <smcv> with a lot of work having been done in between
18:36:45 <spwhitton> I'm sorry to interrupt but I think that we need to move on to the next topic.  Is there anything actionable we can do with this bug?
18:36:45 <ehashman> smcv: I think this is maybe an outdated meme, because Debian moves _much_ slower than most upstreams these days.
18:37:27 <ehashman> spwhitton: I think that someone should volunteer to summarize our consensus, I'm not sure what the next steps for ruling on the bug should be.
18:37:29 <spwhitton> Am I right in my assessment that no-one wants to argue we should override the maintainer as Dmitry wants?
18:38:22 <ehashman> I don't see any value in overriding the maintainer. can't speak for anyone else though
18:38:57 <spwhitton> gwolf: you've spoken the most strongly against vendoring; would be useful for you to answer here
18:39:32 <gwolf> Yes, I was thinking about that -- but as I said, I could not find time to finish reading the thread
18:39:50 <gwolf> so I don't feel I retain enough context on the specific issue...
18:40:14 <spwhitton> In that case it sounds like we should continue to let discussion on the bug occur, hopefully with security team input, and as Elana says, get someone to write up points of agreement in this meeting?
18:40:15 <gwolf> I mean - I can try to do the summarizing in the next few days... but cannot give much of advice right now
18:40:42 <spwhitton> Do we have a volunteer to summarise the meeting?
18:40:46 <spwhitton> ah gwolf?
18:41:30 <gwolf> I will take it.
18:41:40 <spwhitton> #action gwolf to post summary of meeting to bug
18:41:47 <spwhitton> #agreed leave bug open for further input at this tiem
18:41:58 <spwhitton> #topic Follow up from the DebConf20 discussion
18:42:24 <spwhitton> I wasn't at the last meeting but from looking at the logs, it seems there was an aborted attempt to assign people to each of the proposals?
18:43:05 <spwhitton> They would be generic drivers of that proposal as time allows.
18:43:10 <ehashman> spwhitton: we only had like 2 or 3 people at the last meeting, the hope was that we could assign a person to each of the proposals to drive them
18:43:25 <ehashman> so we tabled this until this meeting in the hopes there would be more of us around :)
18:43:33 <spwhitton> Okay.  Assigning people in this way seems like a good idea to me and we do have six of us this time.
18:43:39 <spwhitton> So, volunteers?
18:43:51 * spwhitton finds marga's document again
18:44:25 <spwhitton> #info https://salsa.debian.org/debian/tech-ctte/-/blob/master/talks/rethinking-the-tc.md[5~
18:44:40 <spwhitton> ehashman: you mentioned wanting to continue with proposal #1?
18:44:49 <ehashman> yeah I'm happy to take that one
18:44:59 <spwhitton> #agreed ehashman to drive Proposal #1
18:45:09 <spwhitton> How about proposal #2?
18:45:12 <smcv> work is eating my attention span so I think I have to back away this time
18:46:05 <spwhitton> I could look at #2, but since I am in the middle of job applications and teaching online, progress would be slower than it might be with someone else in charge of it.
18:47:24 <spwhitton> For context, I think we need drivers for proposals #2 and #3, but not #4 and #5 as we sort of abandoned those at DebConf.
18:49:15 <fil> I'll have a go at #2
18:49:30 <spwhitton> #action fil to drive Proposal #2
18:49:35 <spwhitton> great, thanks fil
18:49:59 <spwhitton> proposal #3 is going to require engagement with the community team etc.  I don't really think of myself as much qualified for that.
18:50:06 * gwolf cannot take them right now, sorry - I got an action point I have to find time to fiddle with already!
18:50:40 <spwhitton> bremner is the only unassigned person who hasn't expressed a view yet...
18:50:55 <spwhitton> (since we are two people down today it would be okay not to assign this)
18:51:39 <spwhitton> let's give bremner a minute or two
18:51:57 <bremner> I'd prefer not to make a longish term comittment at the moment
18:52:09 <spwhitton> okay then, let's leave the other three proposals out for now
18:52:19 <spwhitton> #info Proposal #3 lacks a driver at present
18:52:33 <spwhitton> #info Proposals #4 and #5 are probably not going to be driven forwards by TC members
18:52:41 <spwhitton> any other thoughts on this topic?
18:53:00 <spwhitton> would fil and ehashman like to discuss their proposals now in any way?
18:54:56 <fil> I'd probably rather kick ideas around in emails, having not really come up with anything as yet
18:55:02 <ehashman> honestly, I think I'd have to do the research on what needs to change
18:55:12 <spwhitton> that's fine, thank you both for volunteering!
18:55:16 <spwhitton> #topic Recruitment efforts
18:55:42 <spwhitton> marga was to write to d-d-a but she hasn't.  Do we just wait for her here?  Risk is that it's getting late int he year!
18:56:11 <fil> should I start spamming hapless victims again?
18:56:24 <spwhitton> fil: can you do that and also put your script in our repo if you haven't already?
18:57:01 <fil> already done -- I'm very happy to do it and/or walk someone else through it
18:57:19 <spwhitton> I think it's fine to send out some mails with fil's script right now, but does anyone think we ought to wait on the d-d-a mail?
18:57:47 <gwolf> I have to leave, sorry - my family is waiting for me (I told them it was a 1hr meeting)
18:57:51 <bremner> do we know what we want to send? just a generic call?
18:57:58 <spwhitton> bremner: I think so?
18:58:08 <bremner> I'd say we go for it.
18:58:17 <spwhitton> looking at the minutes, yes, generic
18:58:21 <gwolf> fil: Can you share to the private mail alias the list of people you have contacted? (and, if you have some more information on them, also that)
18:58:35 * gwolf disappears swiftly!
18:59:30 <bremner> spwhitton: maybe (someone should) bounce the generic mail to the private list for comments, wait a few days, and send
18:59:54 <spwhitton> I don't have access to mail archives right now due to hardware issues
19:00:02 <spwhitton> are you volunteering mr. notmuch?
19:00:26 <bremner> OK, I guess I can find it. Or at least be ashamed of not finding it.
19:00:27 <fil> gwolf: sure
19:00:48 <spwhitton> #action bremner to circulate last d-d-a mail for comments to private alias, and then probably send it out if no objections
19:01:17 <spwhitton> #action fil to activate his script
19:01:19 <fil> 20 more victims spammed :-)
19:01:21 <spwhitton> #topic Any other business
19:01:29 <spwhitton> any other business?
19:01:33 <bremner> here it is, in case someone wants https://paste.debian.net/1168163/
19:01:39 <spwhitton> fil completed his action item before it was even assigned to him, impressive
19:02:12 <bremner> who is leaving the ctte this year?
19:02:19 <fil> it's either that or forget about it completely ;-)
19:02:20 <spwhitton> fil.
19:02:30 <fil> yup
19:02:50 <bremner> ok, I will do the trivial edit
19:06:20 <spwhitton> #endmeeting