14:59:06 <richard> #startmeeting Tor Browser Weekly Meeting 2023-02-21
14:59:06 <MeetBot> Meeting started Tue Feb 21 14:59:06 2023 UTC.  The chair is richard. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:59:06 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
14:59:14 <RobertMin[m]> Hello!
14:59:16 <richard> pad per usual: https://pad.riseup.net/p/tor-tbb-keep
14:59:57 <Jeremy_Rand_36C3[m]> For those of you who haven't met Robert Min yet, congrats, now you have :)
15:00:12 <ma1> o/
15:00:12 <richard> ahh
15:00:16 <richard> hello RobertMin and welcome o/
15:00:17 <PieroV> Hi, RobertMin[m]! We've heard about your work, very nice!
15:00:43 <RobertMin[m]> Thank you! Really nice to you too!
15:01:49 <richard> ok
15:02:05 <richard> Tor Browser 12.0.3 and 12.5a3 are both released and published
15:02:16 <Jeremy_Rand_36C3[m]> Yay \o/
15:03:23 <richard> we had a couple of problems this time around stemming from the updated dmg toolchain needed to add a custom icon for the mounted dmg (iirc)
15:03:50 <richard> turns out we didn't actually ship 12.5a1 -> 12.5a2 mars for macOS
15:04:08 <Jeremy_Rand_36C3[m]> I hadn't yet heard about the dodged bullet that PieroV mentioned, good to hear that things didn't crash and burn
15:04:09 <richard> so alpha macOS users are getting a very large build-to-build update this time around
15:05:21 <richard> in retrospect we probably could have added 12.5a1 to 12.5a3 incrementals
15:05:23 <richard> oh well
15:06:21 <richard> seems like we're making steady progress on s131 tasks, and we should be building 12.0.3 later this week
15:07:12 <richard> like w/ the January release I'll tentatively schedule a group UX review meeting for probably early next week for us to sit down and poke the build and file bugs
15:08:39 <richard> beyond that lets keep on with the existing s131 tasks in ~Next (I'll go through and promote Backlog issues as appropriate today)
15:08:57 <richard> and obv if there's any question about what the priorities should be feel free to ping me
15:10:31 <richard> PieroV: why don't you take your dicussion point next and then we can hand over to Jeremy+Robert to talk about their project
15:10:40 <PieroV> my point is going to be very long
15:10:48 <PieroV> Does anyone have anything quick to discuss, first?
15:11:12 <Jeremy_Rand_36C3[m]> It sounds like PieroV 's is quite a bit higher-priority than ours FWIW
15:12:16 <richard> I think it's all you PieroV
15:12:16 <PieroV> I'll take it as a no and start with my point then
15:13:07 <Jeremy_Rand_36C3[m]> (yeah I'm kind of curious about this failure mode, sounds like a good war story)
15:13:37 <PieroV> So, for the last alpha we dodged a bullet. It's mainly on me, I had tested a feature only with my dev build, which had stale cache and as a consequence I didn't catch some startup errors that would have made the browser unusable
15:14:01 <PieroV> Luckily, ma1 did a clobber build for other reasons and found the problem
15:14:52 <PieroV> This made me think. And since we're talking about improving QA and the build system, I have a few proposals to possibly avoid similar situation (esp. if their ending isn't happy as this one)
15:15:39 <PieroV> I would like to propose to have stricter dates for what can be merged: close to the release we don't merge anything, unless it has a very good reason to be merged
15:16:13 <PieroV> Instead, we should wait for a nightly to succeed, verify it works as it should, and then use its commit for the alpha, too
15:16:34 <ma1> it's the like the "freeze" pre-release window by mozilla.
15:16:46 <PieroV> Yes, exactly.
15:17:02 <PieroV> Simply put, yes, I propose we start having a freeze window, too
15:17:09 <Jeremy_Rand_36C3[m]> PieroV: by "use its commit", I assume you mean both the tor-browser-build and tor-browser commits?
15:17:26 <PieroV> Jeremy_Rand_36C3[m]: nope, we need to fix things in tor-browser-build
15:17:46 <PieroV> Like the Tor, OpenSSL and Go version
15:17:52 <richard> mmhm
15:18:32 <Jeremy_Rand_36C3[m]> (Since Nightly uses a branch name instead of a commit hash for the firefox project IIRC?)
15:18:32 <boklm> do you mean we merge the "Prepare alpha/stable release ..." MR, and wait for a nightly before tagging?
15:18:33 <PieroV> For Tor we take the latest from the main branch in nightlies, for the rest we use the releases to update everything
15:18:43 <Jeremy_Rand_36C3[m]> PieroV: oh, so you mean just using the tor-browser commit hash?
15:19:17 <PieroV> Especially the tor browser, unless we decide to update dependencies such as Go and OpenSSL before the release preparation
15:19:42 <boklm> I think it's also possible to tag a -build1, and then after checking a nightly, tag a -build2 if necessary
15:19:58 <PieroV> Now that I think of it, we need to bump the Firefox version, too
15:20:10 <Jeremy_Rand_36C3[m]> Yeah what boklm said seems close to what I was thinking
15:20:29 <PieroV> We do it when we do the release preparation, but we need to do it after we have rebased, instead
15:20:58 <PieroV> boklm: Jeremy_Rand_36C3[m]: yes, it could work. Actually we don't even need to tag, until we update the change logs
15:21:07 <PieroV> Just merge an "update" MR
15:21:38 <richard> I think another thing that isn't explicitly called out here is when we do backports from alpha to stable
15:22:09 <PieroV> Yes, I forgot about that, but I think we've been backporting too much
15:22:17 <ma1> right, the storage.sync thing!
15:22:19 <richard> we've had a handful of problems with backports that have happened partially because the release process has called for doing the backports as part of the release after rebasing, so they never actually get tested in stable before publishing
15:22:24 <PieroV> ma1: exactly
15:22:45 <richard> agree on backporting too much
15:22:57 <PieroV> Also, for stable I've written my point 3
15:23:11 <PieroV> (checking nightlies + freeze were points 1 and 2)
15:23:44 <PieroV> Point #3 is: we should build stables asap, to give as long as possible to tor-qa@lists.tpo
15:24:27 <PieroV> I don't know if there are many people checking the binaries we send in that list, but we should give more time to them in case
15:24:56 <henry-x> how long is it currently between building finishing and publishing?
15:24:56 <Jeremy_Rand_36C3[m]> FWIW it seems like the specific failure mode we had this time (browser failed to launch?) should be pretty easy to test in CI, manual QA testing shouldn't be needed if I'm understanding correctly
15:25:16 <PieroV> Also, maybe with the new infra we will be able to do something with CI and get more stable builds. In general, we don't build many stable builds to check it before the actual release
15:25:32 <richard> do you mean that we should have unsigned stable even before we do the rebase?
15:25:41 <Jeremy_Rand_36C3[m]> Obviously manual QA testing is still desirable, but if it's a chemspill fix, CI is going to be a lot less disruptive to getting a release out ASAP
15:25:43 <richard> ^that is also a problem
15:25:58 <PieroV> henry-x: very short, alas. Lately we've released later than Firefox, so we've tried to get everything signed and published asap
15:26:01 <richard> (that being we don't *really* do stable builds in day-to-day dev work)
15:26:52 <PieroV> Jeremy_Rand_36C3[m]: yes, probably some basic automatic CI would have detected this problem
15:27:06 <richard> yeah one problem we currently have is that we just don't know if there will be security backports for android until the official firefox release which is almost a week after we get build tags
15:27:10 <PieroV> I was even thinking to some CI to actually prepare binaries, without us needing to launch them
15:27:29 <Jeremy_Rand_36C3[m]> FYI I have some scripts on Cirrus CI that are designed to test basic Firefox functionality. Shouldn't be hard to adapt them to use Tor Browser, but I would have to put Tor Browser in Transproxy mode because Cirrus infra blacklists Tor traffic (I think due to mining abuse)
15:27:29 <PieroV> So that anyone could possibly get a build, without even asking us devs to build one for them
15:28:05 <Jeremy_Rand_36C3[m]> Let me know if you want me to spend some time on that
15:28:35 <PieroV> Jeremy_Rand_36C3[m]: I think we can do something with our infra, thanks anyway :)
15:28:35 <Jeremy_Rand_36C3[m]> Literally all the scripts do is try to visit a website and save a screenshot, and check if it saved successfully
15:28:46 <Jeremy_Rand_36C3[m]> PieroV: OK, no worries
15:28:56 <PieroV> We have a testsuite, but I don't know if it's actually run
15:29:12 <PieroV> It's very out of date, and hard and hard to run
15:29:31 <PieroV> Because it's based on the old marionette, which is still using Python 2.7
15:29:52 <Jeremy_Rand_36C3[m]> (I'm going to have to rig my scripts to work with Tor Browser anyway since those scripts are designed to test software that needs to work in Tor Browser, but I don't know when the funding for that is going to show up)
15:30:49 <PieroV> I still have two points, besides the CI, which wasn't an expected one but surely an interesting one :)
15:31:04 <richard> :)
15:31:18 <PieroV> One is that we should actually start rotating the rebase role
15:31:23 <PieroV> And also the release prep
15:31:46 <PieroV> I was thinking that richard maybe could do only stable releases for a while
15:32:02 <PieroV> And let other people from the team prepare alphas, so that eventually everyone will be able to prepare releases
15:32:41 <PieroV> We've been talking about that for a while, but we could do it for 12.5a4
15:33:38 <henry-x> how long does release prep take?
15:34:15 <richard> it depends
15:34:50 <richard> that's a good question actually
15:36:41 <PieroV> Well, first, do we include the Tor Browser rebase in it?
15:36:45 <richard> so the entire process generally takes me at least a couple of days of real time, but that's with the various realities of work getting in the way (meetings, MRs, things that demand my attention, etc)
15:36:55 <richard> right^
15:37:13 <PieroV> Because if we include the rebase it needs a second person to review it, which adds time :)
15:38:08 <PieroV> The rebase can be very fast (no conflicts, no MRs that need to go in the middle of everything and cause new conflicts when squashing)
15:38:32 <richard> so the rebase *usually* doesn't take very long, but if there are complicated interactions between patches then it can drag out; if all goes well the act of rebasing should be like an hour tops, and then of course review, any revisions, etc
15:38:58 <richard> but i've had it take several hours when there are weird conflicts
15:39:08 <PieroV> But if you start having conflicts it can take a few hours (like the last time, that we needed to build and test stuff), but no more than 1 day, including other things that need your attention
15:39:25 <richard> yeah 1 day worst case
15:39:35 <PieroV> Rebases on the RR and on the ESR are completely different
15:39:50 <henry-x> ok, and what about the rest of release prep? I have no idea what it involves
15:40:09 <richard> then the 'release prep' (the tor-browser-build patch) is actually pretty quick these days, also on the order of a few hours to go through checklist
15:40:37 <richard> its mostly just updating tags, checking for dependency updates, etc
15:41:04 <richard> (of course all of these things are probably a lot faster if you've done it before, already know the steps and implicit stuff associated with each)
15:41:04 <PieroV> The changelog update used to be very long, but with the scripts it is quite fast now
15:41:24 <richard> yes the changelog update is now more of an editing/update gitlab task then it is a writing task now
15:41:29 <richard> so that's a lot faster
15:42:00 <PieroV> But usually the release prepper also runs the first build (in the build servers, if needed), to fix any problem they find
15:42:17 <richard> yep^
15:42:53 <PieroV> To sum up, I would say not very long for the release we've been doing lately
15:43:03 <richard> my work-flow typically goes from ok we have firefox tags from Mozilla to ok we have a candidate tor-browser-build patch to build in about a work day and then build overnight
15:43:05 <PieroV> But something that can take you from the rest of the tasks you're attending
15:43:31 <richard> but build tags typically come towrd the end of the day EU time so in reality that work-day is split up over two work days
15:43:55 <PieroV> Also, stable releases are somehow longer, because they might involve backports
15:44:11 <richard> ^this though I think that should change
15:44:13 <henry-x> ok. TBH, I'm not that keen on the idea because doing lots of small tasks tends to slow me down. But I'm also wondering whether it make sense that everyone needs to learn the same skill and only use it a few times in the year?
15:44:35 <ma1> did anybody already mention those very helpful gitlahb templates?
15:44:48 <richard> we should do backports after a release so they have a month in the stable branch for us to discover
15:45:01 <PieroV> I think that having a spread knowledge is always a good idea
15:45:09 <ma1> backups are always good
15:45:11 <richard> ma1: yeah all of the release prep is documented in the always updating release prep templates so one is not flying blind
15:45:12 <PieroV> 1. it helps with emergency releases
15:45:22 <Jeremy_Rand_36C3[m]> Besides the other already-mentioned reasons, making sure everyone can do it is useful for improving the bus factor
15:45:36 <PieroV> 2. people can come up with new ideas on how to improve the process
15:45:48 <PieroV> 3. what Jeremy said
15:45:57 <richard> 4. i think that spreading the load a bit will help us avoid burnout in the future
15:46:08 <PieroV> +1
15:46:16 <richard> especially as we add s131 to the list of responsibilities
15:46:25 <PieroV> Also, if we're going to prepare S131 builds it means that we can spread out the 4 builds
15:46:52 <PieroV> And if you know how to prepare a release you also know how to review it
15:47:23 <PieroV> (4 builds i.e., 2 stable + 2 alpha)
15:48:09 <PieroV> Somehow related, and also related to the backports point is my fifth point
15:48:23 <PieroV> IIRC, when I started at Tor, we didn't always use --autosquash
15:48:48 <PieroV> But instead squashed when switching from 12.0.X to 12.0.(X+1) we would have squashes only fixes from 12.0.(X-1)
15:49:04 <PieroV> To make finding bug causes easier, if needed
15:49:24 <richard> that seems like a good idea to me
15:49:26 <PieroV> I.e., we don't squash the changes on the next release, but in the following one
15:49:37 <Jeremy_Rand_36C3[m]> Yeah that seems sane
15:49:42 <PieroV> richard: I think it was your idea originally :)
15:49:54 <richard> i don't recall but it sounds smart so I'll take it
15:50:02 <Jeremy_Rand_36C3[m]> hehe
15:50:25 <PieroV> Maybe we could move the fixups closer to their final destination, without squashing them in
15:50:55 <richard> i like that too
15:52:17 <PieroV> Okay, these were my 5 points, I don't know if richard wants to add more on the 6th (backports), or I'll leave the mike to anyone else for comments in any case :)
15:52:27 <henry-x> is there an easy way to tell that a fixup is from two versions ago?
15:52:52 <richard> the fixups to be merged would occur before the build tag if Im' understanding correctly
15:52:54 <PieroV> henry-x: I think looking at the build tag is an idea
15:53:17 <PieroV> GitLab doesn't show it, but git log does
15:53:57 <PieroV> I mean that GitLab only tells you the tag commit, without showing the tags inline (not even an icon, if a commit has been tagged)
15:53:59 <henry-x> ok, can git rebase only apply --autosquash for a certain range?
15:54:24 <PieroV> You could cherry-pick in two steps
15:54:38 <PieroV> First you cherry-pick anything that needs to be squashed, then you cherry-pick the rest
15:54:42 <richard> yeah exactly
15:54:51 <henry-x> ok, makes sense
15:55:27 <richard> this is already how i do the rebase: start with FIREFOX_102.8 and git chery-pick FIREFOX_102.7..tor-browser102.7-12.0-1-build1 or w/e
15:55:47 <richard> so that could be split up w/o issue
15:55:54 <PieroV> (Hg tags are also super useful)
15:56:01 <richard> anyway we're 3 minutes to the hour\
15:56:07 <Jeremy_Rand_36C3[m]> (Should Robert Min and I wait until next week's meeting for our item, or should we do it in #tor-browser-dev right after this meeting? Leaning toward the former so that we don't sink too much of your time.)
15:56:25 <PieroV> Me and richard have a meeting after this one
15:56:30 <richard> I think we've another meeting after for the tor-browser-release meeting
15:56:40 <PieroV> Yep, meeting afternoon today
15:56:43 <richard> if you want to chat in #tor-browser-dev i'll have the backlog
15:56:54 <Jeremy_Rand_36C3[m]> Let's just do it next week then
15:56:54 <richard> otherwise we can wait until next week if you'd prefer
15:57:06 <richard> ok
15:57:16 <richard> thanks or your time everyone o/
15:57:18 <Jeremy_Rand_36C3[m]> Sorry about the contention for time, usually these meetings don't take up the whole hour
15:57:20 <RobertMin[m]> I also prefer next week!
15:57:31 <richard> #endmeeting