15:00:22 <pili> #startmeeting ESR-Transition 10/03
15:00:22 <MeetBot> Meeting started Thu Oct  3 15:00:22 2019 UTC.  The chair is pili. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:22 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
15:00:27 <pili> hi everyone
15:00:36 <pili> let me create a pad quickly
15:01:20 <pili> https://pad.riseup.net/p/tor-browser-esr-transition-keep
15:01:31 <alsmith> is here to listen
15:01:38 <sysrqb> hi!
15:03:05 <pili> I'm just adding some questions we should figure out the answers to
15:04:17 <pili> in part because we want to get this funded and will need to answer those questions
15:04:48 <GeKo> <- is here
15:04:58 <brade> hi
15:05:07 <pili> and in part because we will need to figure this out to do this transition!
15:05:15 <boklm> did we have an other pad with the same discussion in the past?
15:05:46 <pili> boklm: good point, I think we may have had
15:05:58 * boklm vaguely remembers one
15:05:59 <pili> let me try to dig that up while someone else kicks off the discussion? :)
15:06:29 <sisbell> hi
15:06:42 <sysrqb> okay, so one of the questions we need to answer is how we'll keep up with the release train
15:06:49 <alsmith> pad from our last esr convo https://pad.riseup.net/p/otf-tb-esr-migration-brainstorm
15:06:58 <sysrqb> alsmith++
15:06:59 <boklm> alsmith: thanks
15:07:25 <sysrqb> this is, honestly, a good question
15:08:01 <sysrqb> but right now GeKo and Mike are spending some of their days looking at all of the changes since the last esr
15:08:20 <pili> alsmith: thanks :D
15:08:41 <sysrqb> and that is something we can create roles for, and do it weekly or a few times per month
15:08:48 <GeKo> from what i have fresh in mind right now when doing the feature review
15:08:50 <sysrqb> rebasing and resolving merge conflicts is another issue
15:09:01 <GeKo> i think it takes like 2 days per release for that part
15:09:46 <sysrqb> okay, so with a 4-week cycle, that isn't too bad
15:09:55 <GeKo> yep
15:10:08 <GeKo> i expect a little less work in that case even
15:10:14 <GeKo> as the changes are not as many
15:10:17 <GeKo> (probably)
15:10:18 <mcs> GeKo: That sounds about right, but sometimes more time is needed to dig deeper on some complex change (especially if it is an area of code that is unfamiliar).
15:10:18 <sysrqb> yeah
15:10:18 <tjr> Mozilla has three trains: Release, Beta, Nightly. The intention is that Tor Browser release follows release; and Tor Browser beta follows beta; right?  Would there be a Tor Browser Nightly that follows Nightly?
15:10:18 <tjr> It seems like tracking Beta and Release wouldn't be too hard with regards to ongoing work (e.g. point releases and patches landing in -release and -beta...
15:10:18 <tjr> But it might be 50% or more of someone's time tracking Nightly.
15:10:38 <tjr> Of course not tracking Nightly means that the work when we release (and bump the Beta version) is increased
15:10:53 <tjr> But hopefully the Beta -> Release bump would be minimal
15:11:12 <GeKo> i doubt we have the capacity to track nightly closely
15:11:15 <sysrqb> yeah, and not tracking nightly we don't catch bugs/issues early enough to fix them before Beta
15:11:19 <sysrqb> but that's a tradeoff
15:11:37 <GeKo> and especially to resolve bustage in nightly builds by following mozilla's nightly closely
15:11:38 <pili> I think last time we discussed this we said we would only track 2 trains :)
15:11:47 <sysrqb> yes
15:12:02 <tjr> Okay, cool
15:12:12 <boklm> maybe we could have two nightlies branches, one based on beta, and an other one on nightly
15:12:24 <GeKo> maybe
15:12:27 <boklm> the one based on nightly being more often broken
15:12:32 <sysrqb> we can always add the nightly train later if two trains is simple for us
15:12:37 <GeKo> but starting with the nightly on beta as a goal seems not too bad
15:12:44 <boklm> yes
15:12:53 <boklm> we can still add that later
15:13:25 <sysrqb> this brings us to the question about automated rebasing
15:13:50 <sysrqb> doing the rebasing is easy, but we'll need alerts when it fails
15:14:13 <sysrqb> doing this manually once a week isn't something we can do long-term
15:15:07 <sysrqb> but, overall, i think that shouldn't impact our capacity too much
15:15:35 <sysrqb> unless there are significant merge conflicts or we need to backout a change that landed upstream
15:16:06 <mcs> Do we have enough experience to know that the rebasing will be easy? Sometimes it will not be easy, right?
15:16:21 <tjr> boklm: sent an email about quilt a little bit ago
15:16:24 <mcs> ah, what you just said covers my comment
15:16:36 <alsmith> so part of increasing our capacity to move into this cycle is the autorebasing (which is in O2)?
15:17:27 <sysrqb> i personally think this will be two-fold
15:17:42 <tjr> While the phrase "has been rewritten in perl to make [it] more maintainable" made me simultaneously giggle and cry - I do actually agree that maintaining a series of patches to be applied each time - rather than trying to rebase - sounds good and is worth investigating
15:17:43 <sysrqb> the first part is auto-rebasing, and the second part is testing
15:18:25 <GeKo> alsmith: yes
15:18:55 <sysrqb> tjr: ah, i think when we say "rebase" we really mean "cherry-pcik"
15:18:59 <sysrqb> *cherry-pick, too
15:19:20 <sysrqb> maybe that is part of the confusion?
15:19:31 <sysrqb> at least that was my thought on this
15:19:38 <sysrqb> GeKo: ^ ?
15:19:52 <tjr> Well; when youcherry-pick from branch a to branch b - and there's a merge conflict, the tool will require you to fix it.
15:20:00 <GeKo> no we actually rebase with every esr release right now
15:20:45 <GeKo> like git rebase -i --autosquash mozilla/esr68
15:20:54 <acat> actually i cherry-picked :)
15:21:03 <GeKo> which is nice in some scenarios
15:21:13 <acat> ah, but you mean from now on
15:21:29 <GeKo> like getting rid of all the !fixups and !squashs
15:21:44 <GeKo> and one can reorder the commits as one deems useful
15:22:10 <GeKo> but, yes, rebasing in that sense might not be what we want here
15:22:11 <sysrqb> yeah, autosquash is a nice feature, and we should continue using that :)
15:22:32 <sysrqb> i think we can post-process the branch
15:22:54 <tjr> If you've got a script that (every day) grabs mozilla-beta/master and cherry-picks patches a,b,and c from tor/last-release - when a breaks on day N, it will break on day N+1, and N+2
15:22:54 <tjr> You need to create a new a called a' and stick it in some branch and then cherry pick a'
15:22:54 <tjr> And - as boklm/greg notes - the change from a -> a' is difficult to find - you need to track branches and compare across branches.
15:22:55 <tjr> With the quilt approach, you have a git repo that contains patches a, b, and c - and every day you apply a, b, c on top of mozilla-beta/master.  If on day N, a breaks; you fix up a and commit an update to patch a to the quilt repo - and you can see the history of a as it needs rebasing.
15:23:07 <sysrqb> but for moving out patches onto the newest beta, it seems like we'd want to cherry-pick
15:24:27 <GeKo> tjr: yeah, i think that's a neat approach
15:24:28 <tjr> However - I'm not very familiar with autosquash so if everyone thinks that's a better workflow... :)
15:24:52 <GeKo> and we should explore it for our "rebasing" efforts
15:24:57 <sysrqb> we can use a git branch for creating the patches
15:25:15 <sysrqb> but this is an interesting idea
15:25:26 <sysrqb> boklm: did you have write an email or ticket about this?
15:25:43 <GeKo> i think if that works for a large project like a kernel
15:25:48 <GeKo> it might be for us, too
15:25:49 <boklm> sysrqb: yes, I sent an email a few hours ago on tbb-dev
15:25:54 <antonela> sysrqb, boklm: [tbb-dev] Nightly rebase experiment proposal
15:26:06 <sysrqb> boklm: ah! :)
15:26:11 <sysrqb> heh
15:26:12 <GeKo> (and we even don't have to reinvent the wheel)
15:27:07 <pili> ok, so it seems like we have some idea of the process we will need to take with each new release
15:27:08 <pili> more or less...
15:27:43 <pili> and we should figure out, on average how long that will take us with each release...
15:27:44 <intrigeri> (FWIW in Debian, we use quilt as the storage/transfert format for patches on top of upstream; but many, if not most, people use other tools, such as gbp-pq, to manage this patch series with quilt, because they find quilt too bothersome; I'm part of these people)
15:28:21 <tjr> (A side note - we don't need to talk about right now but is echoing very loudly in my head - is what amount of automation we want to put into Mozilla infrastructure and what into Tor infrastructure.)
15:29:22 <sysrqb> i think that'll depend on what tooling we use
15:29:41 * tjr is now splitting attention between two meetings
15:29:54 <sysrqb> but given we're a little limited on resources, that seems like a good discussion to have
15:29:57 <pili> tjr: that sounds like it could be a good discussion for the tor/mozilla sync? :)
15:30:10 <pili> we wanted to discuss some of this then anyway
15:30:28 <pili> I'm happy to discuss now also if people would prefer though
15:30:53 <sysrqb> i don't think we have all the necessary people, right?
15:31:16 <sysrqb> but we can discuss it on  our side, at least, and get an idea of what we'd want to offload onto Mozilla
15:31:34 <tjr> Sure. My starting point for discussion would be around "What do you want to automate?" and then "How do you want that to behave?"
15:32:26 <tjr> I think any automation that happens in Mozilla infrastructure should probably be considered to be owned and maintained by Tor (and me) and just running there though. So if your build or whatever breaks, it might be disabled automatically, and it'll be up to us to fix it and re-enable it.
15:32:51 <sysrqb> that's understandable
15:33:10 <sysrqb> do mozilla provide this type of infra support for other orgs?
15:33:21 <sysrqb> or would ew be a weird special case
15:33:53 <sysrqb> (if you know, off hand)
15:33:58 <boklm> I think something we could do is pushing our rebased branch every night on Mozilla Try, to run the tests, skipping the patch that changes all our prefs (which breaks a lot of tests)
15:34:12 <sysrqb> yep!
15:34:25 <GeKo> yes, but that's a different can of worms
15:34:46 <GeKo> as tests will still be broken that way
15:35:09 <GeKo> and it might be hard to figure out whether that's due to some nightly feature or whatever
15:35:10 <sysrqb> we'll need to audit every test
15:35:29 <GeKo> i guess it could be part of our myriad of qa efforts, though
15:35:34 <boklm> maybe we can limit the tests that are run to the few that test the most important features we care about
15:35:47 <GeKo> could be
15:36:00 <boklm> as a start
15:36:13 <GeKo> i'd just wary to spent too much time on this
15:36:26 <GeKo> i fear that's just a deep rabbit hole
15:36:32 <boklm> ah yes
15:36:49 <GeKo> i am more in favor of setting up our own integration tests properly
15:36:58 <GeKo> and write more tests for that part
15:37:02 <GeKo> like we did a while ago
15:37:17 <GeKo> we still have that infrastructure running
15:37:32 <boklm> yes
15:37:33 <GeKo> just the tests need to get fixed (again) and then expanded
15:38:24 <GeKo> we are basically avoiding all the infra setup overhead in that scenario
15:38:25 <sysrqb> i think use Firefox tests and Try builds is helpful, but we can disable some of them (or maybe most of them)
15:38:32 <GeKo> sure
15:38:35 <GeKo> no doubt
15:39:00 <sysrqb> at this point we don't know what is broken in Tor Browser compared with Firefox
15:39:20 <GeKo> but there will be hard choices to be made and i would opt to not fix all the tests on try for us
15:39:34 <GeKo> but rather invest in our own infra
15:39:37 <sysrqb> okay, that's fair
15:39:52 <GeKo> but ideally we would have both :)
15:40:24 <tjr> sysrqb: I think Thunderbird has some stuff running in Moz infrastructure but I'm not sure exactly...
15:40:25 <sysrqb> yeah
15:41:34 <sysrqb> tjr: okay
15:42:11 <sysrqb> this is somethign we can thik about
15:42:53 <sysrqb> Monitoring & evaluating: "Describe how the project's success will be evaluated and any metrics that will be collected related to the project's outcomes and impact"
15:43:13 <sysrqb> we can decide on a point where we declare it complete
15:43:39 <sysrqb> maybe after shipping two full releases?
15:44:01 <sysrqb> it seems like this can be a little arbritrary, as long as it's defined
15:44:10 <tjr> It wouldn't be hard to develop milestones. Metrics I'm less sure about.
15:44:30 <sysrqb> yeah, i'm pausing on the metrics part
15:44:33 <tjr> Perhaps "Average time between introduction of a mozilla patch that currently breaks things to time it is dealt with"
15:44:43 <tjr> Current time: 8 months. New time: 2 days - 2 weeks!
15:44:54 <sysrqb> :)
15:45:07 <pili> lol :)
15:45:08 <pili> that is a good way of showing it though
15:45:37 <pili> I wonder if there's something there about less/no reports of tor browser being outdated
15:45:44 <pili> but I'm not sure if we have any metrics of that right now
15:45:52 <sysrqb> we can provide more defined improvements from the ETP portion of it
15:45:53 <boklm> metrics could be how much time per month we spend to keep things working, instead of working on new features
15:46:14 <sysrqb> boklm: ah, that's a good point, too
15:46:17 <pili> boklm: that's a tricky one... it may be that we have less time for new features :)
15:46:37 <sisbell> Metrics are geneally the things the customer cares about but I'm not sure who the customer is here
15:46:52 <GeKo> pili: not really
15:47:03 <GeKo> but i see this from time to time
15:47:07 <pili> we could check the metrics that we had for the TBA project
15:47:16 <pili> get inspiration there
15:47:17 <alsmith> good idea pili
15:47:22 <boklm> in the begining we will spend a lot of time to just keep things working, but after a few releases we should improve and have more time for working on new features
15:47:29 <pili> boklm: I hope so! :)
15:47:59 <sysrqb> (12 minutes warning)
15:48:21 <alsmith> i want to circle back to the ‘sustaining the pace’ question
15:48:30 <sysrqb> (or, maybe let's say 7 minute warning, for those who have a meeting in the next hour)
15:48:37 <sysrqb> alsmith: sure
15:48:47 <mcs> I think overall the new approach will require more engineering resources… forever. But maybe I am pessimistic.
15:48:52 <alsmith> are there other ways we are preparing the team?
15:49:36 <mcs> Regardless, we have to make this transition for mobile at least.
15:49:59 <sysrqb> yeah
15:50:17 <sysrqb> i don't know if this will necessarily require more engineering resources overall
15:50:29 <sysrqb> we just spend the last two months rebasing onto the next esr
15:50:32 <sysrqb> *spent
15:50:47 <sysrqb> that's a lot of engineering resources
15:51:07 <sysrqb> (that didn't take 100% capacity, but it was a large part of it)
15:51:12 <mcs> It is hard to know at this point but I think we will be interrupted a lot to fix things as we go instead of working in bigger chunks…
15:51:22 <mcs> and context switches have a cost
15:51:26 <pospeselr> mcs: +111
15:51:31 <sysrqb> that could be, i agree with that
15:51:49 <mcs> Anyway, what else should we discuss now?
15:51:51 <sysrqb> but, we honestly won't know until we start
15:51:58 <mcs> +1
15:52:00 <sysrqb> and we'll see how this goes with mobile first
15:52:12 <sysrqb> (unfortunatelly, that'll be too late for this proposal)
15:52:35 <sysrqb> alsmith: are there other specific questions we should answer?
15:53:05 <sysrqb> i think the fact that the responsibilities will be distributed over the team, maybe/probably with rotating roles
15:53:14 <sysrqb> is a large change for the team already
15:53:25 <alsmith> we do need to elaborate on the concrete actions — ie, question two in the pad, but i don’t know if there’s really time to do that all now
15:53:26 <sysrqb> and we're getting mentally prepared
15:54:07 <alsmith> ok sysrqb i (am guessing) that staffing/role definition element is something they want to hear
15:54:36 <sysrqb> we can enumerate the concrete changes we're expecting
15:55:13 <sysrqb> they tie into the pieces from question 1, as well
15:55:21 <sysrqb> as to who we'll maintain the pace
15:55:25 <sysrqb> *how
15:57:12 <sysrqb> okay, anything else for the next 3 minutes?
15:58:30 <alsmith> i’m realizing that this wasn’t clearly communicated in question 2 but we need to write more concrete steps for the Activities that start with ‘evaluate’ / ‘set up’ / ‘investigate’ — they are concerned about how we determined our estimations & want to see those details.
15:58:43 <alsmith> maybe thig is something pili & sysrqb could work on?
15:58:55 <sysrqb> alsmith: yeah, sure thing
15:59:02 <alsmith> awesome thanks
15:59:05 <pili> ok
15:59:20 <pili> ok, I think I'll end the meeting here then
15:59:24 <pili> #endmeeting