17:00:02 <hellais> #startmeeting OONI dev gathering 2016-06-27
17:00:02 <MeetBot> Meeting started Mon Jun 27 17:00:02 2016 UTC.  The chair is hellais. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00:02 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
17:00:10 <hellais> hello all!
17:00:13 <anadahz> hi everyone
17:00:45 <agrabeli> heya :)
17:00:46 <hellais> so for this weekly gathering we have the following items on agenda: https://pad.riseup.net/p/ooni-irc-pad
17:01:39 <hellais> so starting from the first item
17:01:48 <hellais> #topic making ooniprobe into a system daemon
17:02:38 <hellais> on this point I guess I mainly wanted to share some of the discussions had with sbs on this topic and see if the conclusion (that is actually just confirming what we would have done) makes most sense
17:02:39 <willscott> hi
17:02:49 <willscott> cool
17:03:45 <hellais> basically during the course of the next month ooniprobe is going to become a system daemon, this will mean that it will be capable of handling the scheduling of the periodic measurements tasks (no more pesky cron jobs), expose a web based user interface and in the future support communicating with the orchestrator
17:03:46 <landers> here
17:05:48 <hellais> given the fact that the long term goal for ooniprobe is to try to move as away from python as possible and possibly replace the measurement primitives with what we are implementing inside of measurement-kit we were wondering if it made sense to make this daemon component something that is enough abstracted to be useful as a launcher for also other measurements
17:06:13 <hellais> that is implement it in some other compiled language, ship it as something different than ooniprobe and make this other thing a depedency of ooniprobe
17:07:13 <hellais> and have it just be something that takes care of launching ooniprobe. Over time features of ooniprobe can be moved into this new component and eventually it can replace it entirely.
17:08:02 <agrabeli> so would users be able to run both ooniprobe and neubot from this interface (and possibly other measurements too)?
17:08:11 <hellais> however in thinking of how this could be done in practice it became apparent that there is quite a bit of overhead associated with starting ooniprobe and running it's event loop. So you can't just do something where you run it as a command every time the web interface says "do this thing".
17:08:38 <hellais> agrabeli: that was the idea.
17:09:41 <anadahz> how far is this from happening?
17:09:42 <agrabeli> hellais: can you pls specify a bit more on the overhead part? are you referring to ooniprobe dependencies?
17:09:59 <hellais> for example ooniprobe when it starts will need to start tor, start the measurement managers and report schedulers, not to consider the memory footprint of running the python interpreter.
17:11:13 <hellais> so in the end the conclusion was that it's probably ideal to stick with the original plan of integrating all of this directly inside of ooniprobe and defining a well done HTTP API that could, potentially in the future be the way in which this other daemon interfaces with ooniprobe.
17:12:19 <landers> what do you mean by the web interface saying "do this thing"? like run a single ooni test?
17:12:58 <landers> and a future http api would let you tell a running oonidaemon to run a single test?
17:13:14 <hellais> agrabeli: I mean that every time you run the ooniprobe command there is a certain memory and CPU cost associated with it. If you need to run 3 different tests you need to start 3 ooniprobe processes that occupy a certain amount of memory for the duration of the run. If instead you could have only 1 ooniprobe process running and tell it to do these 3 things you would not incurr in these costs.
17:15:10 <hellais> landers: yes, the problematic situation that would occur if we weren't to make ooniprobe a daemon and just pilot it as an external process from this launcher is that if I am on the web interface and I want to check if 3 sites are blocked, but for some reason I schedule them individually, 3 processes will be started and that comes with some cost
17:15:20 <agrabeli> hellais: right.
17:16:26 <landers> got it
17:18:29 <landers> and what features of ooniprobe would get moved into the new thing?
17:19:56 <hellais> to clarify the conclusion of this is that this new thing actually should not be done, because of the above mentioned reasons, I just wanted to share the reasoning behind this and know if somebody had a strong opinion for one or the other.
17:20:52 <hellais> landers: what we evaluated to put inside of it are: 1. handling the starting of tor 2. configuration management 3. measurement scheduling 4. daemonisation 5. http interface 6. GUI 7. Orchestration
17:21:01 <landers> instead of making a new thing, the existing thing will become a daemon?
17:21:31 <landers> and grow an http interface and gui?
17:22:07 <hellais> landers: correct.
17:23:13 <landers> sounds good
17:23:33 <agrabeli> I think it would have been nice to have had one interface for various different measurements - I know several operators that would really have appreciated that - though given that there's the overhead issue, perhaps proceeding as you concluded is the most reasonable option atm.
17:25:44 <hellais> agrabeli: yeah, that was one of the main motivations behind it, though the consensus was in the end that we should push towards that only once we have feature parity for all the measurements done by neubot and ooniprobe integrated into measurement-kit.
17:26:42 <willscott> that sounds like a high bar…
17:26:57 <willscott> there are lots of measurements in ooniprobe that we don’t run regularly that have got to be pretty low priority to replicate in measurement-kit
17:27:03 <willscott> like the psiphon and other protocol compatibility things
17:27:24 <anadahz> hellais: the idea sounds very nice but how much development does this require?
17:30:12 <anadahz> willscott: not really related to the discussion but psiphon and other censorship circumvention test *must* be measured regularly, this is currently not happening due to application dependencies
17:30:39 <willscott> sure
17:30:50 <willscott> i feel like we’ll be able to move faster once we only have a single code base for measurements
17:32:35 <hellais> willscott: yes, that's right, though I am not very much convinced of the strategy we have adopted for implementing the CCT in ooniprobe and if I were to implement them in measurement-kit I would go for a solution that doesn't require shipping the actual binary of the software and launching it like we do, but rather implementing a sufficient amount of the protocol to ensure it's working or doing something mor
17:32:40 <hellais> e a-la glasnost
17:32:58 <willscott> agreed
17:33:24 <willscott> i think we’ll end up reimplementing the circumvention tool tests in measurement-kit, and will want to do that by mimicing the protocols
17:34:10 <willscott> but, i’d be okay with switching ooniprobe over to use the mk tests before that was done
17:34:44 <willscott> so i guess a thing we should try to solidify at some point is what the set of tests are that we need working in mk before we’re willing to switch ooni probe over to use that setup
17:35:55 <hellais> willscott: I think the lower bar for that would be having http-header-field-manipulation
17:36:14 <willscott> that’s the only one we’re missing?
17:36:17 <willscott> if so, that’s great!
17:36:23 <willscott> that suddenly sounds much more tractable :)
17:37:09 <hellais> well we have http-invalid-request-line, tcp-connect, dns-injection and web-connectivity is basically done currently in measurement-kit
17:38:20 <willscott> great
17:38:33 <hellais> other missing parts are bouncer communication and support for cloud-fronted backends
17:38:53 <hellais> reporting to http(s) collectors is also done
17:39:32 <hellais> I guess the question here is, does somebody have a good idea of when and how to begin this transition?
17:39:34 <willscott> do we have a ticket for this?
17:39:57 <agrabeli> awesome! but isn't tcp-connect redundant if web-connectivity is included?
17:39:58 <hellais> is it better to do it all in one go or can there be some way of doing it more progressively?
17:40:17 <hellais> agrabeli: yeah, tcp-connect was implemented before web-connectivity existed
17:40:34 <willscott> well, we could start even now figuring out what the dependency and build process looks like
17:40:43 <willscott> and then start having ooniprobe delegate to mk tests for a specific list of tests
17:42:34 <anadahz> hellais: Are we that all ooni-probe tests can be ported to mk and that there are no libraries and dependencies issues?
17:43:02 <anadahz> *sure
17:43:45 <anadahz> MK is a piece that needs an amount of testing and review to reach the usability level of ooniprobe
17:43:59 <hellais> anadahz: what do you mean by "libraries and dependencies issues"?
17:44:50 <willscott> anadahz: i think as we code it as a dependency we’ll hit a bunch of those testing & usability issues
17:44:53 <hellais> worse case we can always make a statically linked MK library and not have to worry about the dependencies, but the dependencies are not so many at the moment.
17:45:38 <willscott> my fear is that if we wait until it’s ‘polished’, we’ll just not prioritize it and it won’t happen anytime soon
17:46:20 <anadahz> hellais: network tests are using specific libraries, many of them already implemented in Python does the same applies to C++11 ?
17:46:46 <hellais> willscott: yes I agree with that. That is also why I was trying to figure out if there was some better way of avoiding to add too much extra code into ooniprobe that will then eventually have to be taken out of it when we finalise the migration.
17:49:38 <hellais> anadahz: the only dynamic linked libraries that MK uses are libssl, libevent and libgeoip
17:51:52 <hellais> anyways the point is that as part of what is to be done by the end of the next month is some sort of GUI for ooniprobe, the question here is does it make sense to try to develop this as another component that is not written in python or not.
17:52:46 <hellais> The pro of doing it as something not written in python is that once we decide to ditch python entirely to replace everything with measurement-kit we won't have to re-write this part.
17:53:08 <willscott> the gui is a web frontend?
17:53:25 <hellais> the contra are that I don't see a simple way of not making this be something that doesn't just launch a bunch of python processes and hence leading to a lot of extra memory usage and it's also longer to develop
17:53:32 <hellais> willscott: yes
17:53:37 <willscott> filed a tracking bug for mk migration: https://github.com/TheTorProject/ooni-probe/issues/558
17:54:19 <willscott> so i guess there are 2 things:
17:54:41 <willscott> 1. the UI itself is something that we can figure out separately of how we implement the actual running of tests in response to user actions
17:55:10 <willscott> and prototyping that ui is something we shouldn’t feel like we’re blocking on, and will hopefully figure out what sort of API we need with ooniprobe
17:55:23 <willscott> like, what sort of status / completion hooks will need to be exposed?
17:56:50 <hellais> absolutely the UI component is something that will survive even the transition
17:57:57 <anadahz> Another point: How long are we going to support Python ooniporbe after the transition?
17:58:49 <agrabeli> hellais: since it's not clear how this could be developed without "launching a bunch of python processes" and since something other than python would (if I understood correctly) lead to a lot of memory loss (thus harder to integrate with other things like neubot, as discussed earlier)...perhaps develop the gui in python?
17:59:14 <landers> why not just javascript talking to the http api?
17:59:46 <hellais> anadahz: I don't think we should support the python version of ooniprobe at all after the transition. I mean it should just replace it.
18:00:38 <anadahz> There a number of devices and usrs using the Python version of ooniprobe we can just stop supporting after transition..
18:01:09 <anadahz> We need make OS packages, installation scripts and guides, help with transitioning
18:01:26 <hellais> landers: yes, that is the plan, however the point I was making before was that if this HTTP API is implemented in python inside of ooniprobe it's code that one day will die, but if we implement it elsewhere then there is some potential for code-reuse.
18:01:39 <landers> oic
18:01:51 <hellais> anadahz: well by no longer support I mean that we will not be making any new release of the python version of it after the transition.
18:02:22 <hellais> anadahz: the OS packages when they are updated will include the new version of ooniprobe and there shouldn't be a problem.
18:03:27 <anadahz> hellais: yes but I guess this will take a bit more time after the finished transition
18:03:51 <landers> i guess it's hard to implement the http api elsewhere without also having to make another api to talk to the python...? o_O
18:04:17 <willscott> yeah, i agree with landers
18:04:26 <anadahz> the part of the dependencies that I mentioned ^
18:05:08 <hellais> yeah I agree, we did throw out some crazy ideas of how this could be overcome (like having an API that is based on STDIN or named pipes), but it was basically just stepping around the problem and adding double the complexity
18:06:12 <hellais> I guess the conclusion that we came to in the end is reasonable, that the web UI should be made so it can be mounted onto something else in the future and should be optimised for code reuse, while the daemon and HTTP API should be done in python inside of ooniprobe even if this will mean that one day it will not exist anymore
18:07:03 <landers> sounds good
18:08:36 <anadahz> hellais: rephrase it better: do you have a sense on when (~) this transition could happen?
18:09:24 <agrabeli> hellais: sounds reasonable. What would the web UI be written in?
18:11:26 <hellais> anadahz: I think it's reasonable to assume this is going to begin sometime in September/October
18:12:00 <hellais> agrabeli: Javascript based on angular.js
18:12:27 <willscott> hellais: is it you doing this prototype?
18:12:50 <willscott> (or first implementation of the ui, not sure if it’s a prototype)
18:13:06 <anadahz> OK so it seems that his will not affect lepidopter for at least the next relase and will continue to work with the Python version of ooniprobe in mind
18:13:11 <hellais> yes I will be developing this next month
18:15:28 <MightyOctopus> [13ooni-probe] 15hellais created 06feature/webui2 (+4 new commits): 02https://git.io/vo7n0
18:15:28 <MightyOctopus> 13ooni-probe/06feature/webui2 14012b9c4 15Arturo Filastò: Add support for starting tests from the HTTP API
18:15:28 <MightyOctopus> 13ooni-probe/06feature/webui2 14f53289e 15Arturo Filastò: [Web UI] Refactoring of web UI...
18:15:28 <MightyOctopus> 13ooni-probe/06feature/webui2 14960f51a 15Arturo Filastò: Lost in commit
18:16:09 <hellais> in this branch there is the skeleton of it
18:18:51 <hellais> I think we are running a bit out of time
18:18:56 <hellais> are there any final remarks?
18:21:34 <willscott> I look forward to seeing a bunch of you in toronto next week!
18:22:22 <hellais> oh yes indeed ! :)
18:22:27 <agrabeli> same!
18:22:53 <anadahz> yay!
18:23:42 <anadahz> https://citizenlab.org/summerinstitute/2016.html
18:25:33 <travis-ci> TheTorProject/ooni-probe (386f539) : The build passed. Diff:https://github.com/TheTorProject/ooni-probe/compare/012b9c457fe6^...386f539652dc, Build https://travis-ci.org/TheTorProject/ooni-probe/builds/140604073
18:26:31 <hellais> well thanks for attending this meeting and for listening and providing useful comments to these questions!
18:27:07 <hellais> I guess next weeks meeting may be skipped since a lot of us are going to be in Toronto
18:27:54 <hellais> or we can perhaps also do something mixed/live/IRC, the mailing list will be informed anywyas
18:28:03 <hellais> #endmeeting