15:02:06 <karsten> #startmeeting metrics team meeting
15:02:06 <MeetBot> Meeting started Mon Oct 14 15:02:06 2019 UTC.  The chair is karsten. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:02:06 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
15:02:12 <karsten> hi sysrqb, acute, irl!
15:03:32 <karsten> https://storm.torproject.org/shared/5h1Goax5eNusxjXJ_Ty5Wl7hFR1uqCReUiN8xdlBG8T <- agenda pad
15:04:29 <karsten> please add topics you want to discuss today.
15:05:32 <karsten> should we start without gaba?
15:06:01 <irl> ok
15:06:06 <karsten> alright:
15:06:10 <karsten> Tor Browser measurements (karsten)
15:06:28 <karsten> sysrqb: thanks for joining us today!
15:06:39 <sysrqb> hi!
15:06:48 <sysrqb> thanks for having me :)
15:06:48 <karsten> we briefly talked about running these measurements ~1.5 weeks ago.
15:06:52 <sysrqb> yes
15:07:02 <reetika> the time is 4pm utc, right?
15:07:14 <irl> 3pm utc
15:07:26 <karsten> current time is 3:08 utc.
15:07:37 <karsten> but you're asking for the community team meeting, reetika? that's in 52 minutes.
15:07:57 <karsten> 15:08 utc to be precise.
15:08:10 <reetika> i am asking about the one, in which stephw will be co-ordinating
15:08:37 <stephw> hi reetika: please send me an email to introduce yourself steph@torproject.org
15:08:55 <karsten> cool. :)
15:09:08 <karsten> sysrqb: so, I wrote you an email with where we are re: tor browser measurements.
15:09:09 <reetika> already sent:)
15:09:37 <karsten> sysrqb: what are your thoughts on setting something up?
15:10:39 <sysrqb> karsten: yes, i got your email and read it, but didn't respond :/
15:11:06 <sysrqb> i think this is something we can work on beginning in the next month or two
15:11:27 <sysrqb> i'll need to discuss this with the team, but it is important that we begin collecting this data
15:11:35 <karsten> when's your next meeting?
15:11:50 <sysrqb> today at...17:30
15:12:55 <karsten> how about we talk again on thursday at 15 utc at our usual meeting time?
15:12:57 <sysrqb> so i can bring this as a discussion point then
15:13:32 <karsten> or we discuss this more via email in the next days?
15:13:39 <sysrqb> this thursday? (17 Oct)?
15:13:48 <karsten> yes, right.
15:14:07 <sysrqb> okay
15:14:16 <sysrqb> i can follow up with an email after our team meeting
15:14:27 <karsten> yes, that would be great!
15:14:30 <sysrqb> and then we can decide if we should discuss this on Tursday
15:14:41 <karsten> sounds good.
15:14:44 <karsten> thanks!
15:14:59 <sysrqb> great! tanks
15:15:46 <karsten> moving on:
15:15:47 <karsten> OpsDocs (irl)
15:16:08 <irl> #30881 is context
15:16:23 <irl> the sysadmins are producing documentation for services, and we have services too
15:16:25 <karsten> this sounds very related to our attempts to have a second operator for all our services.
15:16:47 <irl> we should use similar documentation techniques to ensure that not only can we manage all our services, but so could the sysadmins in a disaster where we are both unavailable
15:17:26 <irl> some html version of this can be found at https://help.torproject.org/ but i'm writing the documents in mdoc in the metrics-cloud repo with a script to update help.tpo
15:17:48 <irl> the first one will be onionoo, as we're changing how deployment for that works
15:17:53 <karsten> can we somehow combine this with automating our services more?
15:18:00 <karsten> ah, okay.
15:18:09 <irl> that's the idea, the document is a manual on how to use the automation
15:18:35 <acute> I would like to do this for onionperf
15:18:54 <irl> and onionperf also uses metrics-cloud automation, so this makes sense as the next thing to do
15:19:22 <irl> once these are done, perhaps karsten you would like to use these as templates for another service
15:19:27 <irl> and eventually we cover all of them
15:19:27 <karsten> how can I help with the onionoo part of this?
15:19:44 <irl> for onionoo once i've done the doc you can review it to add anything that might be missing
15:19:56 <karsten> okay!
15:20:15 <karsten> and yes, having this for onionperf sounds very useful, too. and for the other services after that.
15:20:43 <karsten> alright! moving on?
15:20:48 <irl> yes, that's all for that topic
15:21:08 <gaba> ouch!! I didn't add this meeting to my calendar and i forgot!
15:21:12 <gaba> sorry I'm just connecting
15:21:15 <karsten> hello!
15:21:32 <gaba> hi!
15:21:41 <acute> hi!
15:22:00 <karsten> I'll quickly say something about OONI metadata database, to give gaba the chance to catch up.
15:22:15 <karsten> so, I followed the instructions to mirror that database.
15:22:46 <karsten> this process stopped ~1.5 weeks ago shortly before I left, and I resumed it a few hours ago.
15:22:59 <karsten> apparently it takes many hours for postgresql to import the files downloaded from s3.
15:23:17 <karsten> in summary, no real news today. but a question:
15:23:22 <irl> is the instance in the same region as the bucket?
15:23:41 <karsten> what's the goal here? have a database copy to look for interesting stuff?
15:24:02 <karsten> or is the goal to later have a database copy for metrics-web?
15:24:05 <irl> to have the vanilla tor and bridge reachability data to make visualisations relevant to anti-censorship work
15:24:09 <irl> yeah, for metrics-web
15:24:16 <karsten> it's not the downloading that is slow, it's the processing on the machine.
15:24:21 <irl> we don't need all the tables so maybe we can find a way to exclude some
15:24:41 <karsten> I don't see how we could exclude tables using the described approach.
15:24:49 <karsten> what about downloading a .csv file from them instead?
15:25:02 <karsten> this would be after knowing exactly what we need.
15:25:15 <irl> i think the problem with that approach is that it's not going to scale over time
15:25:26 <karsten> because the file gets bigger?
15:25:31 <irl> yes
15:25:36 <irl> we need a way to get deltas from them
15:25:37 <karsten> but the database gets bigger even faster.
15:26:17 <karsten> downloading a 10M or even 50M file from them once per day shouldn't be a problem.
15:26:24 <irl> if we've learned that it's too big for us to handle, then that's something we will need to discuss with other sponsor 30(?) people to reassess what we can do
15:27:14 <karsten> when I discussed this topic a few years ago with arturo, he seemed to be okay with the .csv file approach.
15:27:18 <irl> we blocked last time on needing the ooni team to implement this interface for us to download the data in some way
15:27:29 <irl> they didn't have the time to make that interface and it didn't already exist
15:28:26 <karsten> hmm, which part didn't they have time for?
15:28:29 <gaba> irl: wonder if this changed this time. is this something that can't be acceded throug their API (sorry if it is a stupid question)
15:28:51 <irl> the api doesn't produce aggregates
15:29:04 <irl> or even summaries
15:29:12 <irl> it only gives you measurements last time i looked
15:29:19 <karsten> but they're producing aggregates somewhere.
15:29:21 <karsten> right?
15:29:40 <taffo> Hello
15:29:53 <karsten> it would be sufficient to produce a .csv file once per day and store it somewhere accessible for a web server.
15:30:10 <irl> it would be, but that didn't go anywhere
15:30:13 <karsten> if the tricky part is deciding how to produce that .csv file, we can probably help.
15:30:18 <irl> maybe we can ask them again
15:30:38 <irl> i hadn't looked closely enough at the mirror instructions, it's not postgres mirroring, it's importing a dump every time
15:30:57 <karsten> it might even be that importing this dump now and writing some queries helps.
15:31:12 <karsten> yes, it's importing dumps every time, and that's crazy for production.
15:31:17 <irl> yeah if we can write the query for them then maybe that's a lot easier
15:31:28 <irl> postgres can even write the csv file too
15:31:35 <karsten> right!
15:31:50 <irl> ok, let's see if we can do that and ask them again if they'd give us an endpoint for it
15:32:15 <irl> that seems a good next step
15:32:15 <karsten> alright, I'll keep the import running and then take a look at the database.
15:32:18 <irl> cool
15:33:12 <karsten> moving on?
15:33:14 <irl> ok
15:33:18 <karsten> Reviews of #19332 and #31941 (karsten)
15:33:22 <karsten> this is just a ping.
15:33:41 <irl> these would have happened last week but i destroyed by desktop by trying to reproduce the jetty bug locally
15:33:58 <irl> an apt upgrade went terribly wrong and it took me a day to get a working environment back
15:33:59 <karsten> uh oh. :/
15:34:21 <karsten> okay, great, you're aware of these.
15:34:32 <irl> the (main) lesson learned was: don't install firefox and firefox-esr together because they will fight over which gets to open links
15:34:45 <karsten> heh
15:34:45 <irl> i'll get to them this week
15:34:54 <karsten> which one wins?
15:35:03 <irl> firefox-esr apparently
15:35:14 <karsten> alright!
15:35:19 <karsten> Funding proposal status (karsten)
15:35:27 <karsten> gaba: is this submitted?
15:36:06 <gaba> not yet :( Grants teams was concerned that the amount is much more than what we useually ask
15:36:17 <gaba> so we are having meetings with people at Mozilla
15:36:22 <gaba> to go over the proposal first
15:36:31 <karsten> okay.
15:36:38 <gaba> one meeting this week and then I will talk with them at mozilla festival in london next week
15:36:54 <karsten> let us know if you need anything.
15:37:00 <gaba> yes, thanks
15:37:13 <karsten> moving on:
15:37:15 <karsten> Exit scanner (gaba)
15:37:32 <karsten> you asked somewhere about torbel.
15:37:35 <gaba> Yes. I moved things around in the roadmap for us to start working on this sooner rather than later
15:37:40 <gaba> Yes. Sent you a mail
15:37:52 <irl> exit scanner is next up after onionoo migration is complete
15:37:59 <karsten> so, TorBEL is probably 5+ years old and was never in production.
15:38:03 <irl> i would have liked to have already started on it
15:38:05 <gaba> s30 tasks can wait a little more so this one has highger prioritiy
15:38:23 <gaba> ahh, ok
15:39:04 <karsten> irl: when do you expect the onionoo migration to be done?
15:39:32 <irl> end of the week unless we hit some problem like a bug in systemd that breaks stuff in ways you can't predict
15:39:35 <irl> it's unlikely
15:40:00 <irl> everything is basically done except the priv sep
15:40:01 <karsten> okay. please let me know when you're running into similar problems like the missing .jar.
15:40:08 <irl> yeah will do
15:41:05 <karsten> gaba: we should finish the onionoo stuff first, because stopping this now and picking it up later would be sad.
15:41:33 <gaba> ok
15:41:36 <karsten> irl: what about the exonerator thing. let me find the ticket.
15:41:44 <irl> it also somewhat tells us how exit scanner works because we would deploy it in the same way
15:41:47 <karsten> #31071
15:42:10 <karsten> did you expect anything else from that in preparation of the exit scanner rewrite?
15:42:43 <irl> i'm not sure what the question is
15:43:35 <gaba> if there is anything that anybody can do to help the exit scanner rewrite later?
15:43:40 <karsten> ah, it's about missing exit scanner data and how that affects exonerator results.
15:44:03 <karsten> I think you had some hope that resolving this one would help us with the exit scanner rewrite later.
15:44:26 <karsten> the only remaining parts are UX, translations, and then deployment.
15:44:34 <irl> i can't remember anything about it, but i can update the ticket if i do
15:44:37 <karsten> except if you had hoped to learn anything else.
15:44:40 <karsten> ok. :)
15:45:01 <karsten> so, let's talk more about the exit scanner on thursday?
15:45:06 <irl> sounds good
15:45:47 <karsten> added to the agenda.
15:45:54 <karsten> Roadmap - how are we doing? (gaba)
15:46:28 <irl> no changes for me, things there are still accurate
15:46:34 <karsten> no planned changes for me, either.
15:46:41 <gaba> ok
15:47:00 <acute> same for me
15:47:10 <karsten> cool!
15:47:17 <karsten> EOL
15:47:24 <karsten> anything else to talk about today?
15:48:08 <irl> nope
15:48:15 <acute> not from me
15:48:28 <gaba> one more thing. There is a possiblity to have a cybersecurity policy fellow (from hewlett foundation). Let me know if this is something that can be useful...
15:49:20 <irl> gaba: can you send details in an email?
15:49:28 <karsten> heh, that would also be my question.
15:49:43 <gaba> yes... I need to check for more info but I will and send it to you.
15:49:48 <karsten> thanks!
15:50:11 <karsten> alright, thanks, everyone. talk to you on thursday, same time. bye! o/
15:50:49 <karsten> #endmeeting