16:59:21 <hellais> #startmeeting
16:59:21 <MeetBot> Meeting started Mon Nov 30 16:59:21 2015 UTC.  The chair is hellais. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:59:21 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
16:59:31 <hellais> so folks, let's get started. Who is here?
16:59:34 <landers> here
16:59:41 <poly> hello
17:00:03 * sbs here
17:02:02 <hellais> I guess we can start with some report backs
17:02:23 <hellais> what occupied your time last week from the ooni multiverse
17:02:24 <hellais> ?
17:03:00 <poly> in the network meter universe I worked with sbs on merging improved regex parsing into NM
17:03:35 <poly> sbs provided me with feedback on a couple of points and I've discussed and implemented the changes
17:03:52 <poly> they should be pushed to master anytime now
17:04:05 <poly> EOU
17:04:10 <landers> i've been writing ooni-sysadmin shell scripts. got done with the "deploy backend" one and nearly with the "deploy a scheduled ooni-probe bridge reachability test" one.
17:04:19 <landers> eof
17:05:37 <hellais> I've been working on making changes to the ooni-pipeline to work with the latest version of AWS and re-ran it against all historical data up until date. I also worked on moving the OTF funding proposal forward and talking to joe and brennan to coordinate the OONI data explorer/API work to be done with them.
17:05:58 <hellais> EOF
17:06:35 <sbs> Making progresses towards measurement-kit 0.1.0 and reviewed poly's pull request
17:06:37 <sbs> EOF
17:07:02 <MightyOctopus> [13measurement-kit] 15AntonioLangiu pushed 1 new commit to 06fix/coveralls: 02http://git.io/vB70A
17:07:02 <MightyOctopus> 13measurement-kit/06fix/coveralls 14f725418 15Antonio Langiu: Fix
17:08:13 <hellais> if we are done, are there some topics you would like to bring to the table for discussion?
17:08:55 <poly> the next step for Network Meter I think is deciding how to parse the data for visualization
17:09:14 <hellais> poly: what sort of visualisations are you thinking of?
17:09:17 <poly> at the hackathon, we agreed to use D3/C3 for visualization
17:09:37 <poly> visualization for the results for the plugin outputs within NM
17:09:42 <hellais> poly: is there a ticket about it? If not it would be good to open it and invite feedback and discuss it there.
17:09:56 <MightyOctopus> [13measurement-kit] 15bassosimone pushed 1 new commit to 06fix/coveralls: 02http://git.io/vB7uq
17:09:56 <MightyOctopus> 13measurement-kit/06fix/coveralls 14e072ec6 15Simone Basso: build matrix: try #1
17:09:57 <hellais> mockups of what the visualisations would look like are also very useful to describe it
17:10:38 <poly> there is indeed a ticket: https://github.com/measurement-kit/network-meter/issues/31
17:11:01 <hellais> poly: I would suggest starting with some concrete examples of what specific questions you plan to answer visually and then move towards generalising it for all
17:11:14 * poly takes notes down
17:11:29 <hellais> poly: excellent! It would be good to add to the ticket some notes about what is the produced output that is planned to be visualised
17:11:59 <hellais> I imagine that in the future most of the heavy lifting would be done in the backend API and in fact the collectors already return a report ID that is added to the report (at least for OONI results)
17:12:27 <sbs> hellais: what do you mean by heavy lifting?
17:13:28 <hellais> sbs: I mean the analysis of the result. For example: the http_requests test is currently purely detected on the client side, but in the future shall use the baseline control for comparison since Tor is not an ideal control channel
17:13:37 <hellais> same thing goes for the DNS consistency test
17:14:21 <hellais> for OONI tests at least, you are interested in the end in a boolean flag that is an indicator for censorship and I think we have learned that doing this entirely in the client is not feasable especially given the areas where we plan to deploy
17:14:40 <hellais> s/censorship/anomalous network activity/
17:14:59 <willscott> hellais: how is progress on pipeline-ng for generating those baseline controls?
17:15:05 <MightyOctopus> [13measurement-kit] 15bassosimone pushed 1 new commit to 06fix/coveralls: 02http://git.io/vB72i
17:15:05 <MightyOctopus> 13measurement-kit/06fix/coveralls 14b605ac4 15Simone Basso: check value of variables
17:15:47 <hellais> willscott: we had some issues with the pipeline and some of the machines running the baselines. We expect to have a working advanced prototype of the api and pipeline by end of year
17:16:14 <willscott> are you leading up work on that if I have more questions / want to help?
17:16:22 <sbs> hellais: understood -- I guess that the OONI apps for mobile and network meter share similar design goals in terms of what is to be presented to the user and in terms of interaction with the backend, then
17:16:35 <hellais> moving into making them a useful baseline for other test developers will require a bit more time and I am thinking that there should be a way to apply authorship to reports in order to have good baselines
17:16:36 <anadahz> Hi #ooni
17:16:50 <hellais> willscott: yes, please direct questions to me :)
17:16:54 <anadahz> sorry bit late
17:17:06 <hellais> landers and another developer are also working on this
17:17:17 <hellais> sbs: yes I believe so
17:17:45 <hellais> willscott: what do you think of the plan of adding some sort of signature to the report headers to attribute authorship of the measurements.
17:18:29 <hellais> willscott: because we also don't want to end up considering as a baseline results that come from the same ASN as a user, but that are not run under our supervision
17:19:28 <MightyOctopus> [13measurement-kit] 15bassosimone pushed 1 new commit to 06fix/coveralls: 02http://git.io/vB7VK
17:19:28 <MightyOctopus> 13measurement-kit/06fix/coveralls 14ab419fb 15Simone Basso: try with sudo equal to false
17:19:35 <willscott> i'm worried that signature validation on every report multiple times would be slow. The thing should be validated once on input, and shouldn't be touched within the analysis code, presumably
17:21:01 <hellais> willscott: yeah that is a fair point, that is why I think it should go hand in hand with the changing of the reporting mechanism. If we reduce the overhead of having to de-serialise YAML and convert to JSON in the pipeline we probably have enough CPU to do the validation
17:21:27 <willscott> having a way to know that two reports come from the same client sounds good though
17:21:34 <hellais> I am also thinking that probably a full signature is not needed for the actual content of the reports, but perhaps something weak like a checksum
17:21:58 <hellais> the alternative would be to make authenticity be applied to the reports by specific dedicated collectors, that are end to end authenticated with trusted probes
17:22:34 <hellais> also it wouldn't necessarily have to be that much CPU intensive as most reports would not include the signature, but only the ones that are run for control
17:23:14 <hellais> If there isn't a ticket already open about this I should probably open one to discuss this furthere there
17:23:22 <willscott> sounds good
17:25:53 <MightyOctopus> [13measurement-kit] 15bassosimone pushed 1 new commit to 06fix/coveralls: 02http://git.io/vB7rQ
17:25:53 <MightyOctopus> 13measurement-kit/06fix/coveralls 14c1e6742 15Simone Basso: comment out code for testing -- hello ooni meeting :)
17:26:05 <anadahz> bnvk: HI!
17:26:27 <anadahz> bnvk: (sorry for the capital) is simon around?
17:27:35 <MightyOctopus> [13measurement-kit] 15bassosimone pushed 1 new commit to 06fix/coveralls: 02http://git.io/vB7oB
17:27:35 <MightyOctopus> 13measurement-kit/06fix/coveralls 147520ce9 15Simone Basso: more commenting of .travis.yml
17:27:39 <hellais> anadahz: simonv3 is MIA
17:27:52 <hellais> I suspect some timezone related issues
17:29:05 <hellais> https://github.com/TheTorProject/ooni-probe/issues/438
17:29:10 <hodgepodge> Hi, I'm just going to casually interject here with regards to checksums, we could use a fast hash function like BLAKE2. If we're using YAML within the pipeline, we should probably switch to JSON until the final reporting stage to reduce the intensity of the YAML type conversion.
17:29:13 <hellais> willscott: ^^ I made a quick braindump of what we said
17:30:46 <hellais> hodgepodge: yeah that could be a good idea, though we probably would have to do some benchmarks of this and figure out some best scheme. It's probably a matter of finding some ideal compromise
17:31:07 <hellais> hodgepodge: can you add that as a comment to the ticket?
17:34:48 <anadahz> on my side: reviewed PRs and help on the OpenVPN and psiphon OONI tests, add some features and bug fixes on lepi#
17:35:07 <anadahz> sbs: shall we contact OTF ?
17:37:33 <sbs> anadahz: I sent emails to everyone and updated the shared document on the current status, some persons haven't replied yet and I'd wait some 3-5 more days
17:37:50 <hellais> what is this about?
17:38:09 <anadahz> hellais: reimburrsment for the mini hackthon
17:38:25 <hellais> ah, shouldn't it go through Aspire?
17:39:14 <hellais> anyways we can talk about this at a later time
17:39:52 <anadahz> hellais: it was just a reminder for sbs/me
17:40:28 <hellais> anadahz: sounds good :)
17:40:40 <hellais> so is there more to be discussed?
17:40:53 <hellais> otherwise we shall go into next steps
17:42:05 <hellais> I plan to be submitting the OTF proposal end of day tomorrow and ping them about it. Then I will be working on the ooni-pipeline code and following joe to get him up to speed with what is needed. I will also be following simonv3 that will be working on the GUI related stuff and writing some code here and there. Summary: main focus: * ooni-pipeline code * ooni-api code
17:42:10 <hellais> EOF
17:43:05 <sbs> I will continue the process of releasing measurement-kit 0.1.0, help with proposal
17:43:19 <sbs> EOF
17:43:51 <landers> so we've got the add_headers_to_db task running, next is the spark_apps task
17:44:02 <landers> after those, we can get the API running
17:44:14 <landers> that's roughly it, right hellais ?
17:44:39 <hellais> landers: correct
17:44:45 <landers> then EOF
17:47:06 <anadahz> will finish polishing OONI documentation, bring up some ooni probes, finish work on some reports, fi#
17:47:42 <anadahz> nalize lepidopter user documentation
17:47:45 <anadahz> EOF
17:48:09 <poly> I'll brainstorm requirements and plans for visualization within NM
17:48:12 <poly> EOF
17:51:53 <anadahz> oh i 'll prepare the OONI report for last month
17:52:16 <anadahz> in case you have anything to share...
17:53:16 <anadahz> you could add it on a pad
17:53:58 <hellais> anadahz: yeah please add it to a pad
17:53:59 <anadahz> ^ just for Oct. 2015
17:55:08 <anadahz> hellais: ok
17:55:15 <hellais> ah ok, I shall do the one for November in that case
17:56:52 <anadahz> ok need to go..
17:57:01 <anadahz> thanks everyone for attending
17:57:03 <anadahz> afk
18:00:11 <hellais> indeed thanks for being in the meeting
18:00:12 <hellais> oh
18:00:25 <hellais> simonv3: it seems like there was a timezone management issue
18:00:35 <hellais> we just ended the meeting
18:00:51 <hellais> #endmeeting