16:02:12 <hellais> #startmeeting OONI gathering 2016-10-03
16:02:12 <MeetBot> Meeting started Mon Oct  3 16:02:12 2016 UTC.  The chair is hellais. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:02:12 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
16:02:32 <hellais> great so it seems like we have a pretty well outlined agenda today
16:02:53 <hellais> thanks agrabeli and everyone else that helped sketch that out
16:03:41 <hellais> so let's start from item 1.
16:03:57 <hellais> #topic ooniprobe v0.3.0: Integration of ooniprobe with measurement-kit
16:04:23 <hellais> I assume this is actually ooniprobe v3.0.0 or maybe v2.1.0 or anyways something > 2.0.0
16:04:48 <sbs_> yes, this would probably be v2.1 at least
16:05:12 <sbs_> anyways, here's how I think we can proceed
16:05:56 <sbs_> 1. during Seattle, hellais and I had a long discussion regarding scriptability of MK and during this discussion we also discussed how to integrate MK in OONI and wrote a prototype for that
16:07:03 <sbs_> 2. one of the things we learned while making a prototype is that the optimal thing would be to be able to use the existing ooni-probe machinery and _only_ call the part of MK that, given the input and settings, runs the measurement and returns the report entry
16:07:24 <hellais> (see: https://github.com/TheTorProject/ooni-probe/commit/22fb1b310156fd713827178c6c6567c52946aaa0)
16:08:08 <sbs_> 3. (not totally unrelated) today I released v0.3.{0,1} of MK (and v0.3.2 is on the way) and I think this stable release could be the one we use to implement the integration
16:08:26 <sbs_> 4. there are no debian packages yet, and I need to ask irl for help
16:08:59 <sbs_> 5. one can already have MK on macOS using this tap: https://github.com/measurement-kit/homebrew-measurement-kit
16:09:28 <sbs_> 6. this is the repository containing the Python extensions for MK: https://github.com/measurement-kit/measurement-kit-python
16:10:05 <sbs_> (cont) in this repository today I implemented the approach discussed with hellais and described in 2: https://github.com/measurement-kit/measurement-kit-python/blob/master/examples/tx.py
16:10:14 <sbs_> (while at it, I also fixed a bug with Twisted code)
16:11:20 <sbs_> there is basically a function (see https://github.com/measurement-kit/measurement-kit-python/blob/master/measurement_kit/tx.py#L23) that invokes the test on a background MK thread but what Twisted sees is just a deferred
16:12:00 <sbs_> the `input_` is a URL, `settings` is a dict mapping strings to scalar types eventually converted to strings by MK
16:12:36 <sbs_> the `entry` passed to the callback is a python object parsed from a JSON (specifically it should be a dict)
16:12:58 <sbs_> this dict contains only the `test_keys` part and hence the caller should do something like:
16:13:12 <sbs_> real_entry['test_keys'] = that_dict
16:13:35 <sbs_> I think this summarizes the plan and status of the integration; hellais anything you would like to add and/or suggest?
16:14:19 <hellais> I think that is a pretty exaustive summary of where we are at.
16:15:17 <sbs_> hellais: good
16:15:18 <hellais> I guess the only thing is that we also need to ensure that the per test (and not per-input) lookups are done only once even when calling the function that takes a single input as we discussed in SEA
16:15:43 <hellais> but that is quite into the weeds
16:16:41 <hellais> I think another thing that we should try doing for testing the integration of MK is re-writing the test templates using MK. That would give us an idea of how much effort it would be to swap those out.
16:17:04 <sbs_> hellais: uhm... the per-test lookups are not performed by the specific function processing the input (i.e. no probe ip resolution, no geoip) because those in MK are performed by the "driver" class (i.e. OoniTest)... so the answer is probably that ooni-probe should perform those (as I guess it probably does?)
16:18:35 <sbs_> hellais: in theory the complexity of rewriting the test templates using MK is similar to rewriting the tests: in both case we're talking about functions of the type: func(inputs..., callback(entry))
16:19:35 <sbs_> hellais: however, speaking of code duplication, I'd rather not reimplement just the templates, because that would duplicate, e.g., the web-connectivity logic and I don't think that would make our code base more maintainable, isn'it?
16:19:39 <hellais> sbs_: hum, that is also one way to go abouts it. Currently the only thing that represents an issue is the lookup of the probes resolver that is only done as part of the web connectivity test. I can see some benefit in doing that for every test and moving that logic outside of the test itself
16:21:09 <hellais> sbs_: well the point of supporting templates via MK is that people that want to keep using the old way of scripting things using the ooniprobe test API can do so, but with the measurement-kit engine underneath
16:21:47 <sbs_> hellais: right, for the time being I guess we can do that in the C++ implementation of the Python extension (it's not complex to do that there) and then do this change in ooni-probe?
16:22:17 <sbs_> hellais: fair point that one of allowing people to script using the MK engine, alright, then, I'll create a ticket for that
16:22:23 <hellais> one of the things that I believe make using MK superior to using twisted for networking stuff is that the errors are logged in a much more verbose and detailed way.
16:22:51 <hellais> sbs_: it's not too complex to move this into ooniprobe actually
16:23:18 <hellais> (I mean the client resolver lookup)
16:23:30 <hellais> anyways I think we will be discussing this more as we go forward
16:23:39 <hellais> probably also when landers is around
16:23:43 <hellais> landers ^^
16:23:59 <hellais> does anybody have anything to ask or add to this topic or can we move onto the next item?
16:24:31 <agrabeli> I don't have anything to add
16:24:32 <sbs_> hellais: okay, for the time being: https://github.com/TheTorProject/ooni-probe/issues/616
16:24:42 <agrabeli> sbs_ and hellais: thanks for the update
16:25:19 <sbs_> and https://github.com/measurement-kit/measurement-kit-python/issues/2
16:25:51 <hellais> ok great
16:25:54 <hellais> so next item
16:25:57 <hellais> #topic Belin meetup schedule, is there any clarity regarding the dates
16:26:35 <agrabeli> so in our previous meeting I suggested end of October-early November
16:26:46 <agrabeli> but I think the precise dates are subject to availability and funding
16:27:02 <sbs_> I cannot the 29/10 but can starting from 30/10
16:27:30 <agrabeli> we don't appear to have a specific budget for this, so we would either have to try to scrape some cash for this & go for the cheapest options, or request for funding (in which case, it would be harder to determine the dates today)
16:28:30 <agrabeli> sbs_: great, I think we can aim for the first week of November, starting from Monday 31st October
16:28:45 <agrabeli> but again, all of this is subject to available budget/funding
16:29:12 <agrabeli> darkk: one of the main reasons why we want to have this meeting is so that we can meet you irl. which dates would work best for you?
16:30:19 <darkk> agrabeli: any variant withing Schengen area is OK for me for almost any dates this autumn
16:31:01 <agrabeli> darkk: great
16:31:01 <hellais> darkk: how much in advance do you need to have the date fixed in order to ensure a smooth visa process?
16:31:45 <darkk> hellais: I already have valid Schengen visa till mid-2017, IIRC
16:32:57 <hellais> ah, great.
16:33:40 <hellais> ok so I guess the next steps for this are to reach out to these potential funders that could provide support for the event and if not get in touch with Tor to see how we can get some budget for this figured out
16:34:17 <sbs_> makes sense!
16:34:18 <hellais> do we have more to cover on this topic or can we move onto the next?
16:34:34 <darkk> ok, so at this moment we're not even sure that Berlin is the exact location, so it basically gives clear answer to the question =)
16:35:40 <hellais> the location is going to for sure be in schengen and it's going to be with 70% chance Berlin and 30% chance Rome
16:36:48 <agrabeli> Ideally we would aim for the first week of November, and if that's not possible (due to funding constraints), then it will be moved to a later date in November
16:36:54 <sbs_> hellais: bayer munchen, then? :-P
16:37:18 <agrabeli> Berlin is the ideal choice because many other OONI contributors live there, plus we could get venues with 0 cost
16:37:33 <agrabeli> but if we have a very limited budget, then we would go for Rome, since most of us live in Italy
16:38:16 <agrabeli> darkk: We expect to be able to provide you with specific information by the end of this week, or by early next week
16:38:28 <hellais> yup
16:38:37 <hellais> ok let's move onto the next item
16:38:41 <hellais> #topic ooniprobe v2.0.0: do we have a release date that is tightly coupled with some public event
16:39:03 <hellais> I'm not sure how wrote this question. What is meant by "some public event"?
16:39:03 <sbs_> hellais: what public event?
16:39:13 <hellais> lol
16:39:14 <agrabeli> to my knowledge, we don't have the release coupled with a public event, but we do have it coupled with the release of an associated blog
16:40:02 <hellais> agrabeli: can you clarify the meaning of "public event" within this context?
16:40:09 <darkk> hellais: IIRC, you were mentioning something like that that you wanted to release 2.0.0 at some conference, and while digging into #612 blocker deadlock I wanted to refresh my mind regarding the date :-)
16:40:27 <darkk> so, obviously, I don't remember that correctly
16:40:42 <agrabeli> hellais: I didn't suggest this agenda item
16:41:18 <agrabeli> oh, perhaps this event is the OTF Summit that hellais is going to?
16:41:51 <hellais> darkk: ah, ha I see now. So if we would like to target announcing it at some sort of conference in a talk and call for people to install it.
16:42:30 <hellais> we don't really have any big talks lined up in the next month so there isn't a particular event that this should be mapped to
16:42:55 <agrabeli> hellais: but ofc you could also announce it at the OTF Summit?
16:42:57 <hellais> that said the OTF summit is on the 17th of October and it would be nice if we could have it be released by the time that happens
16:43:38 <hellais> though the format of the event is not quite focussed on people giving talks and the like, but more about small breakout groups and workshops
16:43:53 <agrabeli> what are the next steps in terms of doing a stable release?
16:44:36 <hellais> agrabeli: I think the main blocker is issue #612
16:44:44 <agrabeli> hellais: ah ok
16:45:17 <hellais> I also got some feedback on some simple improvements that can be made to the web UI, that I would like to get done before the stable release is made
16:45:45 <sbs_> hellais: are the IM tests part of v2.0.0?
16:45:51 <agrabeli> hellais: great! perhaps we can also make some changes based on the feedback from the user risk session at TorDev
16:45:53 <hellais> sbs_: no
16:46:00 <agrabeli> (they are minor changes, I think)
16:46:47 <agrabeli> hellais: are there plans to integrate the IM tests in the web UI?
16:47:02 <hellais> agrabeli: yes, but not as part of 2.0.0
16:47:35 <agrabeli> I would probably need to create some documentation for these tests, but it would be good to iterate on where they will exist/how users can run them, because that will determine the documentation
16:49:00 <agrabeli> should we move onto the next item of the agenda?
16:49:31 <hellais> agrabeli: it would be great if you could summarize in a ticket the suggested improvements discussed during the risks session. I can't find notes about those suggestions.
16:50:00 <agrabeli> hellais: I've written the notes and I'm going to upload them today on the Tor wiki. I'll also create a ticket for them.
16:50:19 <hellais> agrabeli: great!
16:50:29 <hellais> ok, let's move onto the next item
16:50:36 <hellais> #topic Tor block in Egypt on 2016-10-02, do we want to add technical analisys of Tor fingerprinting?
16:51:07 <hellais> so according to some people on IRC egypt has begun blocking tor users.
16:51:42 <hellais> looking at the graphs on metrics, though, it seems like the disruption is not as extensive as one would expect: https://metrics.torproject.org/userstats-relay-country.html?start=2016-09-05&end=2016-10-03&country=eg&events=off
16:52:11 <darkk> Actually, I don't see any drop at metrics. May it be a data lag?
16:52:38 <hellais> darkk: yeah, that's what I was meaning to say
16:53:02 <darkk> https://metrics.torproject.org/userstats-relay-country.html?start=2016-09-20&end=2016-10-05&country=eg&events=off -- data lag confirmed :)
16:53:27 <darkk> seems, the 1st is the last datapoint on the chart
16:54:18 <agrabeli> the last data point is not much higher than 24th september though
16:54:18 <hellais> ah, yes, I thought the lag was only of 24 hours, but apparently it's greater
16:54:28 <hellais> https://metrics.torproject.org/userstats-relay-country.html?start=2016-09-30&end=2016-10-02&country=eg&events=off
16:55:14 <darkk> "anyway, I've no Egyptian shell yet, s
16:55:23 <hellais> anyways if we can get some vantage point inside of egypt or have somebody run some measurements, it would be good to at least collect some data about it
16:55:57 <hellais> ideally also with some pcaps
16:56:30 <hellais> if we feel like doing some more experiments then one way to go abouts it is running the daphne test with the tor handshake to bisect the censor fingerprint: https://github.com/TheTorProject/ooni-probe/blob/master/ooni/nettests/experimental/daphne.py
16:56:43 <agrabeli> we plan to release a short report on network anomalies in Egypt soon. If we're able to find evidence of Tor being blocked, that would make a stronger case.
16:56:46 <hellais> I am not sure however if this test still works properly as I haven't run it in very much time
16:57:41 <hellais> daphne is a bit complicated to setup, basically you need to have a clean pcap of how what you want to replay looks like on both the client and some control backend
16:58:05 <darkk> so it's probably not a blocker for Egyptian report, but if we'll get some rootshell (to ooni-probe, learn & pcap) we can publish something more interesting. What should we consider a cutoff date?
16:58:49 <darkk> daphne looks wonderful, seems, it may reduce amount of scapy labor :)
16:59:12 <hellais> yeah, I wouldn't consider this a blocker to releasing the blogpost on the general disruptions in egypt
16:59:27 <hellais> though both of these should in general have lower priority than the 2.0.0 release
17:01:04 <hellais> also I should point out what are the priorities for this month: https://github.com/OpenObservatory/roadmap/blob/master/sponsors/narval.md#month-6---october
17:01:40 <hellais> once the release is done we should begin the work on the pipeline reworking
17:01:41 <agrabeli> The main reason why I think it would be nice to also include evidence (if it exists) on Tor being blocked is because it would make a more powerful story than the one we have now. That said, that's not necessarily a blocker for the release of the post.
17:02:53 <agrabeli> hellais: indeed
17:03:40 <agrabeli> darkk: on the egyptian case, is it really the case that ISPs are redirecting users' connections to porn sites to advertisements? Or is this something that is happening on the side of the websites?
17:04:24 <hellais> I should also say that I will be still be not super available until the 5th of October as I am still in travel. In general I will be reading emails and logging on from time to time to IRC, but may not be super responsive.
17:04:47 <darkk> agrabeli: I have no explicit evidence, but I think I can gather some as it's still possible to reproduce that in a reliable way according to source. But that's a good question.
17:05:00 <darkk> agrabeli: I've not questioned that claim. Shame on me :)
17:07:24 <agrabeli> darkk: This week I'm busy with other reports, but I'd be happy to help write up the Egyptian post so that we can publish it asap. That said, I haven't done the research behind this and so I am not in a good position to know exactly what is happening... As hellais mentioned, the priorities are the ooniprobe 2.0 release and later the pipeline stuff, but if you have a clear idea of some of the key findings on Egypt, please share them with me
17:08:05 <agrabeli> hellais: ack
17:08:46 <hellais> ok, do we have any more topics to talk about during this weeks gathering?
17:08:58 <agrabeli> I don't think so
17:09:05 <darkk> agrabeli: ok, so it gives us a ~week to gather something interesting __if__ we'll network resources for that (that's still not clear)
17:09:21 <agrabeli> darkk: kk
17:09:43 <hellais> sounds good
17:10:07 <hellais> well if we don't have anything left I guess we can end this just 10 minutes overtime :P
17:10:17 <agrabeli> hellais: sounds good
17:10:36 <hellais> thank you all for attending! Have a great week!
17:10:39 <hellais> #endmeeting