18:01:12 <spwhitton> #startmeeting
18:01:12 <MeetBot> Meeting started Tue Jan  9 18:01:12 2024 UTC.  The chair is spwhitton. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:01:12 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
18:01:16 <spwhitton> #topic Roll Call
18:01:18 <spwhitton> Sean Whitton
18:01:18 <Myon> Christoph Berg
18:01:19 <tumbleweed> Stefano Rivera
18:01:21 <Emperor> Matthew Vernon
18:01:32 <helmut> Helmut Grohne
18:04:46 <spwhitton> #topic Recruitment
18:06:03 <roehling> Timo Röhling
18:06:17 <spwhitton> roehling: please join jitsi
18:07:49 <spwhitton> #topic merged-/usr
18:08:06 <mjg59> Whoops, I'm also here
18:08:12 <helmut> let me briefly summarize the stuff on d-devel where I asked the ctte for advice
18:08:57 <helmut> in essence, you can navigate yourself into a situation where you schedule removal of a package with dpkg --set-selection and then unpackaing another where that unpack would happen despite a conflict
18:09:25 <helmut> as a consequence aliasing can cause file loss of the just unpacked file (i.e. no maint script is run in between unpack of the lost file and loss of the file)
18:10:00 <helmut> we can mitigate such loss in postinst, but still have a window where the file is actually missing, which is a policy violation when it happens to e.g. zcmp from gzip
18:10:38 <Myon> does that only apply to removals and --set-selections?
18:10:43 <helmut> as long as we do not have mutual, versioned conflicts (or conflicts+breaks), this situation has not been reproduced when using apt-based tools and we believe that apt won't run into this
18:11:17 <helmut> Myon: it happens when you schedule the a conflicted package for (temporary or non-temporary) removal and the unpack the other package
18:11:42 <Myon> that sounds rare enough, I'd bet people only use --set-selections for "hold"
18:11:43 <spwhitton> I have only an intuitive sense of what scheduling for removal, but not actually removing, means.
18:12:01 <helmut> it allows dpkg to remove a package as it sees fit
18:12:20 <spwhitton> right.
18:12:22 <Myon> spwhitton: echo "bla remove" | dpkg --set-selections
18:12:26 <helmut> then when you unpack the other package, it notices the conflict and proceeds due to the intent of removal, then performs the actual removal
18:12:49 <spwhitton> sounds like it would be helpful during manual dependency resolution
18:13:10 <spwhitton> which might be exactly the sort of situation merged-/usr puts you in.  is that the concern?
18:13:48 <helmut> that's an interesting question. quite definitely DEP17 causes more conflicts than usual and it is plausible that you'd have to navigate yourself out of a mess of conflicts
18:14:07 <spwhitton> esp. if you have some pkgs not from the debian archive
18:14:48 <Myon> I'd only be concerned if there is a real-world workflow that has some chances of being seen in the wild
18:15:16 <Myon> I doubt people use --set-selections to remove packages
18:15:27 <helmut> in order to hit the actual loss you need two ingredients: a) a mutual conflict or a manual echo "foo remove" dpkg --set-selections + dpkg --unpack b) aliasing between the conflicting packages
18:15:28 <Myon> and then it would also have to be on a problematic package I guess?
18:15:53 <mjg59> If this were documented as unsupported in the release notes, would that be sufficient for us to consider it resolved? (Not necessarily asserting that that's the right outcome)
18:16:07 <spwhitton> mjg59: I think it would, yeah.
18:16:28 <helmut> we intend to document that people shall run something like dpkg --verify post upgrade
18:16:52 <helmut> packages that are broken due to loss can be repaired by being reinstalled
18:17:02 <mjg59> If so I'd maybe propose that we document it early, publicise that, and see if we hit any pushback
18:17:06 <helmut> systemd and gzip will mitigate their loss in their postinst
18:17:23 <Emperor> amusingly, dpkg --verify currently picks up some oddities on my (much-upgraded) desktop
18:17:24 <mjg59> I don't think we have enough information to be able to judge whether it's a realistic problem
18:17:27 <helmut> so even if you hit the loss with these packages, the consequences are mitigated by the time postinst runs
18:17:30 <roehling> how likely is it that the temporary file loss will break an upgrade so badly that you can no longer restore it afterwards?
18:17:41 <spwhitton> mjg59: right.  we need some way to hear from people with more unusual systems.
18:17:57 <Myon> about how many packages with extra postinst code are we talking?
18:18:11 <helmut> roehling: we may loose poweroff/halt/reboot etc and zcmp/zgrep/...
18:18:56 <helmut> also affected are dhclient and something else (without postinst mitigation at the time of this writing)
18:19:06 <helmut> cryptsetup -> askpass
18:19:24 <helmut> I guess askpass should be mitigated as it may render systems unbootable
18:19:30 <mjg59> helmut: Can we automatically identify packages where there's a risk?
18:19:46 <helmut> mjg59: I have such a list
18:20:07 <mjg59> And is there a meaningful risk of unfixable outcomes for packages outside essential? I guess cryptsetup is an example of that.
18:20:26 <mjg59> Hmm that does make things more complicated
18:20:43 <helmut> keep in mind that - especially due to y2038/time64 - we'll be adding a lot more conflicts
18:21:21 <helmut> and each of them (albeit not being mutual) may be affected on its own (though no known reproducer when performed with apt)
18:21:34 <spwhitton> helmut: what do you think our options are?
18:21:47 <Myon> what's the actual question for us?
18:22:18 <helmut> I'm asking for advice for how we can deal with the unforseen consequences of the DEP17 strategy where conflicts do not work in the way aniticpated
18:22:49 <helmut> especially whether we consider the risk of file loss severe enough to warrant more complexity
18:22:52 <mjg59> It feels like there are three possibilities:
18:23:00 <mjg59> 1) We revert /usrmerge
18:23:06 <mjg59> 2) We document it as unsupported
18:23:14 <mjg59> 3) (2) but also we mandate mitigations
18:23:32 <helmut> 4) we defer further moves and modify dpkg in trixie then move forward with a dpkg that no longer looses stuff
18:23:36 <Myon> how intrusive/bad is the postinst code?
18:23:41 <mjg59> Ok true
18:23:50 <Myon> does it fix the problem?
18:23:55 <mjg59> I think (1) is not a realistic outcome?
18:24:08 <Myon> 1 is not going to happen
18:24:10 <helmut> Myon: for systemd it's reinstating a few symlinks, for gzip I put the shell scripts into postinst verbatim
18:24:30 <Myon> is it custom code tailored for each package?
18:24:40 <Emperor> presumably 5) we do usr-merge the way the dpkg maintainer thinks is correct
18:24:48 <helmut> Myon: in essence, one has to ship all affected files twice (e.g. hardlink) and the restore them in postinst. and for essential packages we violate policy with no way to not violate
18:24:54 <helmut> Myon: yes
18:24:59 <Myon> eww
18:25:01 <Myon> how many more packages would need these fixes?
18:25:44 <tumbleweed> I guess we have a range of options for how many packages to mitigate
18:25:45 <helmut> basically every time you say "conflicts" and either file is aliased. ~100
18:26:26 <spwhitton> helmut: do you suspect that there are ways other than via --set-selections that this problem could break a system?
18:26:27 <helmut> so for very central stuff like systemd and gzip, I felt adding a postinst mitigation was worth the effort, but I haven't done it for more yet
18:26:55 <roehling> Intuitively, the most critical cases are the file losses which render the system unbootable and/or without network access
18:26:59 <Myon> evil idea: could we stuff "dpkg -i $this_package.deb" into the postinsts?
18:27:02 <tumbleweed> I think you're right about just mitigating essential (and cryptsetup)
18:27:20 <helmut> spwhitton: say you have packages a version 1 and b version 1, then you upgrade to a version 2 which conflicts b (<< 2) and b version 2 which breaks a (<< 2). then apt will do those dpkg --set-selectiosn
18:27:31 <Emperor> Myon: I think dpkg's lock prevents such things
18:27:50 <Myon> yeah I would expect that, but maybe it would Just Work
18:27:50 <spwhitton> helmut: so apt needs patching too}?
18:28:04 <helmut> spwhitton: the idea is that we won't du mutual conflicts
18:28:13 <tumbleweed> but we haven't been able to reproduce that problem with apt?
18:28:24 <spwhitton> helmut: but people have local packages.
18:28:27 <helmut> tumbleweed: apt reproduces the problem as above.
18:28:34 <helmut> spwhitton: yes, that's a risk
18:28:42 <spwhitton> surely the apt maitnainers will want to put in a fail-safe
18:28:55 <spwhitton> so that apt aborts if it detects the situation
18:29:09 <helmut> spwhitton: we'll be upgrading with bookworm's apt
18:29:14 <Myon> I'd think local packages wouldn't be involved in these cycles (or not move files or other exclusion criteria)
18:29:15 <spwhitton> oh right of course
18:29:57 <spwhitton> so, the fact that apt can do this means that just documenting it as unsupported is not adequate.
18:29:59 <tumbleweed> although we do encourage upgrading apt early (IIRC)
18:30:05 <helmut> we're here in this situation, because we ignored the dpkg maintainer saying that what we do is broken. now it's broken. (:
18:30:07 <tumbleweed> (or we have in the past)
18:30:20 <spwhitton> hm -- we could document that the mutual conflicts aren't allowed?
18:30:34 <Myon> I'd opt to say "manual postinst fixes for important things" and release note for the rest
18:30:46 <helmut> spwhitton: they're only "not allowed" if they involve aliasing, right? we do have mutual conflicts otherwise
18:31:15 <spwhitton> helmut: right, I mean something saying if you have packages satisfying such-and-such, an upgrade is not supported.
18:31:33 <spwhitton> and we can helpfully add that packages in the debian archive are all fine.
18:31:34 <tumbleweed> brb
18:31:48 <helmut> I don't think it's hard to avoid those mutual conflicts inside debian. we can just monitor and file bugs/patches
18:32:22 <helmut> avoiding them merely means that you cannot experience the problem via apt (to be more precise: we do not know a way to reproduce a problem, but we cannot say for sure that you cannot reproduce it)
18:32:58 <helmut> you can still fiddle with dpkg and experience the loss
18:33:17 <helmut> also no clue about upgrading with cupt
18:33:24 <helmut> or dselect
18:34:03 <spwhitton> helmut: I feel like we need some options to think about.
18:34:24 <tumbleweed> .
18:34:29 <helmut> well, we basically have two options only: a) accept some breakage or b) don't merge in trixie
18:34:46 <Emperor> :(
18:34:49 <spwhitton> I mean, there are a lot of different possibilities under (a) surely
18:34:53 <Myon> not b)
18:35:18 <Myon> we should be discussing about "how much a)", not the rest
18:35:26 <spwhitton> Myon: delaying it is not the same as reverting it.  it's only the latter which is beyond consideration.
18:35:28 <Myon> ... what spwhitton says
18:35:31 <helmut> for instance gzip will violate policy and this policy violation is technically unfixable in trixie
18:35:57 <Myon> spwhitton: I know. Still 1) bad press 2) maintainers will not want to have to care
18:36:49 <Myon> helmut: how about copying zgrep etc to /usr/local/bin for the duration of the upgrade?
18:37:02 <Myon> then if it gets lost, its still callable
18:37:14 <helmut> doing so is a policy violation
18:37:29 <Myon> do you want answers or debate policy?
18:37:36 <mjg59> Also potential for user-owned copies
18:37:47 <helmut> we're mitigating one policy violation with another?
18:37:56 <Emperor> I think we should not sit lightly on policy violation(s)
18:38:02 <Myon> yeah. We are in bug-land here. Saying some workaround isn't perfect doesn't help at all
18:38:21 <Myon> I'd take any number of policy violations over "system is unbootable"
18:38:32 <tumbleweed> how bad is the gzip policy violation? I saw from the thread that it woludn't be usable immediately after unpacking?
18:39:14 <helmut> it lasts from unpack of gzip (after echo zutils remove | dpkg --set-selections ) to gzip.postinst
18:39:40 <helmut> i.e. very temporary
18:39:48 <spwhitton> helmut: the mere fact of a policy violation can't settle the issue, right?  it would have to be that we think someone is relying on that particular point of policy and we can't mitigate the damage from that assumption being violated it with something in the release notes.
18:40:15 <spwhitton> Myon: I'd judge the press quite differently -- I actually think it makes us look pretty good when we delay because we're being extra careful.
18:40:25 <helmut> well, any package can assume that zcmp works at all times and we violate that property
18:40:46 <helmut> but then we also had upgrades where perl would fail to run if unpacked in inopportune order
18:41:27 <spwhitton> I'm not sure we have enough information to judge how bad it is.
18:41:30 <Myon> helmut: I'm confused by the "or" in "mutual conflict or --set-selections" at 18:18Z - is --set-selections a prerequisite for the problem, or are mutual conflicts enough without selections?
18:41:57 <helmut> Myon: mutual conflicts cause apt to issue the problematic --set-selections
18:42:10 <Myon> oookay that sounds much worse
18:42:19 <Myon> that wasn't clear at all
18:42:26 <tumbleweed> idle thought: Can apt tell us enough about its upgrade plan that we could validate it pre-upgrade?
18:42:55 <tumbleweed> would it be feasable to create a pre-upgrade script to check that the install should be upgradable?
18:43:21 <helmut> david also made it quite clear that he does not want to revert to the older behaviour of apt where it did not use --set-selections as doing so would make some upgrades fail
18:43:43 <spwhitton> tumbleweed: right, I was thinking we could provide some way to find mutually conflicting packages.
18:43:51 <helmut> spwhitton: I have
18:44:00 <helmut> currentl 0
18:44:15 <tumbleweed> I mean for a machine weird non-Debian packages installed
18:44:16 <spwhitton> I mean a script for people to run in case they have local packacges
18:44:19 <Myon> then let's make sure it stays 0
18:44:21 <mjg59> helmut: You said that you believed apt wouldn't run into this in the real world - do we have a strong explanation as to why?
18:44:55 <helmut> mjg59: no. only the weak one "we weren't creative enough to beat apt into reproducing it"
18:45:14 <mjg59> Mm, that's not super reassuring
18:45:18 <helmut> agreed
18:45:22 <spwhitton> helmut: we don't have much time left in this meeting.  do you have some more specific questions to ask of us?
18:45:38 <spwhitton> or, do we have some requests for specific informatoin to make of helmut?
18:45:38 <helmut> can I move forward?
18:45:42 <mjg59> If we're going to make a decision I think we could do with a firm statement about ht's happening
18:46:02 <mjg59> But I think there's a reasonable chance that we'll end up having to make a decision on this at some point
18:46:03 <spwhitton> helmut: move forward with filing bugs with the postinst mitigations?
18:46:06 <helmut> i.e. is the problem severe enough to pull brakesß
18:46:16 <spwhitton> well, that seems unclear.
18:46:38 <Myon> If dumat can make sure the number of problems stays 0 before the release, that's good enough (plus maybe a note for local packages, but they would have to move files around and conflict and... maybe unlikely)
18:46:46 <helmut> spwhitton: in two weeks, y2038/time64 will upload a lot of moved libraries and that'll add a lot of conflicts
18:47:05 <Myon> (I'm again confused about how that 0 and the 100 from above relate)
18:47:16 <helmut> Myon: "number of problems stays 0" only applies to upgrading with apt. the problems are still there when fiddling with dpkg
18:47:29 <Myon> I'd forget dpkg
18:47:30 <helmut> Myon: 100 is the non-mutual conflicts
18:47:42 <Myon> and these aren't a probem?
18:47:48 <tumbleweed> but only if you do the remove --set-selections fiddle
18:47:53 <helmut> yes
18:48:00 <tumbleweed> and that's not something *I've* ever had to do or run into before
18:48:11 <tumbleweed> so it does seem a little arcane
18:48:23 <tumbleweed> however, we are expecting lots of posibility to need it, this time around
18:48:27 <helmut> we're assessing risk and cost of mitigation in essence
18:48:34 <spwhitton> helmut: but no-one will be trying to upgrade to trixie for a year and a half
18:49:01 <Myon> to me, it sounds unlikely enough so we can proceed
18:49:07 <helmut> so whenever we hit the actual loss problem for real, there is two two cases:
18:49:28 <helmut> a) the conflict was added for a diversion mitigation. we cannot avoid the conflict. we can only mitigate the loss in postinst
18:49:36 <Myon> and we could still tell people that have to resort to dpkg when the apt upgrade fails, to dpkg -i *.deb twice
18:50:00 <helmut> b) the conflict was added for a move between packages and aliasing change. we can avoid the conflict using protective diversions and thereby remove the problem entirely
18:50:48 <helmut> that b) case incurs non-automatable maintainer script complexity though, so I prefer conflicts for the sake of not getting maint scripts subtly wrong
18:50:58 <spwhitton> helmut: we're almost out of time, and we're rather deep in details.  how about you file a bug asking for TC advice?
18:51:39 <helmut> sure
18:52:08 <spwhitton> sorry we couldn't help more in this meeting.  but I think you've convinced us that we need to think about it.
18:52:10 <helmut> I already have one takeaway though: for every conflict that involves files relevant to boot, we should add the postinst mitigation
18:52:33 <helmut> i.e. if moving forward, that's a minimum requirement.
18:52:41 <spwhitton> right.
18:52:55 <spwhitton> I'm going to move on to AOB
18:52:59 <spwhitton> #topic Any Other Business
18:53:07 <spwhitton> does anyone have anything else?
18:55:29 <mjg59> Nothing here
18:55:56 <Emperor> not me (and I have to vanish imminently)
18:55:59 <spwhitton> okay, many thanks all.
18:56:01 <spwhitton> #endmeeting