13:59:00 <hellais> #startmeeting OONI Community gathering 2018-06-25
13:59:00 <MeetBot> Meeting started Mon Jun 25 13:59:00 2018 UTC.  The chair is hellais. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:59:00 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
13:59:21 <hellais> looks like the IRC bridge is down
14:00:37 <hellais> xxx
14:00:42 <hellais> ok we are back online
14:00:50 <slacktopus> <agrabeli> @t_ooni-slack because Slack has cool emojis :penguin_dance::party_parrot:
14:01:05 <slacktopus> <hellais> @agrabeli meetbot is running so we can start the community meeting
14:01:10 <slacktopus> <agrabeli> great
14:01:24 <slacktopus> <hellais> IRC is supported for legacy reasons :P
14:01:25 <slacktopus> <agrabeli> Hello everyone. Welcome to the June OONI Community Meeting! :slightly_smiling_face: :ooni:
14:01:40 <slacktopus> <agrabeli> I'm Maria & I work with OONI. Who do we have with us here today?
14:02:40 <slacktopus> <hellais> Hi I am Arturo
14:02:56 <slacktopus> <tafiti> Moses!
14:03:02 <slacktopus> <khairil.yusof> o/
14:03:03 <slacktopus> <agrabeli> You can also just wave with your handles :slightly_smiling_face:
14:03:19 <slacktopus> <t_ooni-slack> :wave:
14:03:53 <slacktopus> <agrabeli> As a reminder, please add the topics that you'd like to discuss during today's meeting here: https://pad.riseup.net/p/ooni-community-meeting
14:03:56 <slacktopus> <berhantaye> Hello, Berhan here
14:04:07 <slacktopus> <agrabeli> And feel free to introduce yourselves asynchronously as you join :slightly_smiling_face:
14:04:20 <slacktopus> <agrabeli> Hello @tafiti @khairil.yusof @berhantaye :slightly_smiling_face:
14:04:45 * kushal is here for the first time :)
14:04:58 <slacktopus> <khairil.yusof> kaeru here from Malaysia
14:05:09 <slacktopus> <agrabeli> @kushal ooooh welcome to the OONI-verse :slightly_smiling_face:
14:05:27 <slacktopus> <darkk> :wave: :ooni: :flag-ru: :nerd_face:
14:05:51 <slacktopus> <agrabeli> So let's get started with the first agenda item?
14:06:04 <slacktopus> <agrabeli> topic #1 Sub-categories and tags for citizenlab/test-lists
14:06:13 <slacktopus> <agrabeli> Would the person who proposed this like to share some words?
14:06:21 <slacktopus> <khairil.yusof> ok
14:06:25 <slacktopus> <khairil.yusof> :slightly_smiling_face:
14:07:19 <slacktopus> <khairil.yusof> I believe everyone is familiar with the Citizen Lab test-lists right? https://github.com/citizenlab/test-lists/tree/master/lists
14:08:03 <slacktopus> <khairil.yusof> for those that are not, for the OONI desktop app and raspberri pi image, this is the source url for the daily global and local lists, but also for the ooni mobile app
14:08:57 <slacktopus> <khairil.yusof> so when we run a complete daily web connectivity test, or random web connectivity test, we will be testing against these lists
14:09:46 <slacktopus> <khairil.yusof> other than urls, it also includes other details such as categorization, date added, notes
14:10:42 <slacktopus> <sarath_ms> :wave:
14:11:34 <slacktopus> <khairil.yusof> so the reason I brought up this topic is that, for those of use who use the OONI data to write reports, the broad categorization and existing metadata isn't very helpful for those of us writing reports to automatically filter data based on these lists and categories
14:12:23 <slacktopus> <khairil.yusof> As a result, the categories have been forked into other lists, or maintained elsewhere
14:12:32 <slacktopus> <agrabeli> @khairil.yusof Indeed, that's a good point. The existing categories are very broad and thus require us to manually check each website to better understand what it's about (particularly when found to be blocked).
14:12:35 <slacktopus> <khairil.yusof> the issue here https://github.com/citizenlab/test-lists/issues/264
14:13:28 <slacktopus> <hellais> I think that specifically having tags for adding richer categorization of URLs it probably not something that should be added to the citizenlab test-list repo, but I would suggest keeping track of that in some way separate from the test-lists repo.
14:13:59 <slacktopus> <hellais> The reason for this is that we have seen that it’s already hard enough for people to add a very broad and high level category code as the ones currently in the test-lists and asking them to make it even more specific can lead to confusion
14:14:16 <slacktopus> <agrabeli> @hellais what if tags are optional?
14:14:25 <slacktopus> <hellais> The idea behind having an extra tags column was primarily for adding automatically generated tags that are useful to housekeeping of the repo
14:14:33 <slacktopus> <hellais> Perhaps calling them tags is a bit misleading
14:15:28 <slacktopus> <agrabeli> @hellais what if we have an extra column (to be filled-out optionally), that allows for more detailed categorization?
14:15:31 <slacktopus> <hellais> Also tags in the sense of category tags would actually require something that is more than just a boolean flag, as what is being suggested in that ticket, but would require having at least another level of nesting
14:16:20 <slacktopus> <khairil.yusof> one suggest I have is a sub-category (optional but standardised)
14:16:22 <slacktopus> <hellais> > what if we have an extra column (to be filled-out optionally), that allows for more detailed categorization?  I would consider this a regression based on the evolution of the citizenlab test-list. The reason to migrate over to a more high level set of category codes was precisely to make them more broad and easier to understand.
14:16:42 <slacktopus> <hellais> See for context: https://github.com/citizenlab/test-lists/issues/27
14:17:12 <slacktopus> <khairil.yusof> understand only at broad level for filing
14:17:17 <slacktopus> <agrabeli> @hellais right, but we could require that broad/high-level categories are filled out, while sub-categories are optional, no?
14:17:38 <slacktopus> <khairil.yusof> I'm not a fan of tags in the main list either
14:17:40 <slacktopus> <darkk> What's the purpose of the tags? Citizenlab is just a place to track URLs to be tested regularly. They don't cover OONI Run links, so report-writing should rather be solved with https://www.similarweb.com/ as not all URLs are categorized in citizenlab repo.
14:18:19 <slacktopus> <hellais> I also suspect that the category code you need to make things really useful are so many and vary so much from country to country, that whatever taxonomy you come up with for, let’s say malaysia, may not work as well for other countries
14:18:21 <slacktopus> <darkk> (similarweb or similar categorization service)
14:18:50 <slacktopus> <agrabeli> How about we just include "sub-categories"/detailed descriptions in the "notes" column?
14:19:00 <slacktopus> <hellais> I think you are much better off maintaining this mapping of categories on your own end, rather than having to start a process for standardizing a new set of sub-categories
14:19:21 <slacktopus> <hellais> > How about we just include “sub-categories”/detailed descriptions in the “notes” column?  That will add extra clutter to the test-lists and make things confusing and harder to maintain
14:19:45 <slacktopus> <khairil.yusof> not knowing why they're there is also confusing
14:19:52 <slacktopus> <hellais> Also my understanding is that the need is to have something that is automatically parseable, which would not work in the case of a freeform text field like the notes one
14:20:04 <slacktopus> <khairil.yusof> to maintain
14:20:26 <slacktopus> <hellais> > not knowing why they’re there is also confusing  You mean not knowing why a particular website is in the test-lists?
14:20:32 <slacktopus> <khairil.yusof> yes
14:20:39 <slacktopus> <khairil.yusof> there are some from 2014
14:20:53 <slacktopus> <hellais> Well that is a problem that we were hoping would be solved with the notes field, but we have seen not that many people actually add a note
14:21:07 <slacktopus> <hellais> If the goal is to explain to a human why a URL is in the test list I think the notes field is adequate
14:21:18 <slacktopus> <khairil.yusof> that is one thing that we could standardise on.. on how to use the notes field?
14:21:23 <slacktopus> <agrabeli> @khairil.yusof the ones from 2014 are the old URLs used by the Open Net Initiative. When that project ended, they handed over their test lists to the Citizen Lab who added them to the github test lists.
14:21:48 <slacktopus> <khairil.yusof> anyways, a use case might be helpful here on why sub-categories is useful
14:22:06 <slacktopus> <hellais> > that is one thing that we could standardise on.. on how to use the notes field?  The goal of the notes fields was precisely to not have a standard and allow it to be freeform for everything that doesn’t fit into the above and to expose it to humans (not machines).
14:22:41 <slacktopus> <hellais> What we have seen is that the standardisation process takes many years to actually converge (ex. to update the category codes it took overall ~2 years)
14:23:11 <slacktopus> <khairil.yusof> when we want to test for elections there are certain sites which are specifically for elections, eg. political party, election results, candidates
14:24:24 <slacktopus> <khairil.yusof> if there was some sort of standardization we can compare across countries/regions
14:24:28 <slacktopus> <hellais> My suggestion, as mentioned, would be to keep track of this sort of level of detail in some sort of internal mapping between the CLSI test lists and your own testing lists.
14:25:06 <slacktopus> <khairil.yusof> OK, but it would be good to have an upstream coordination on the lists
14:26:00 <slacktopus> <eiko> Okay just in o/ (was scrolling through what happening here)
14:26:27 <slacktopus> <khairil.yusof> so maybe for now, external list with people interested to join?
14:26:43 <slacktopus> <khairil.yusof> an join on url as key?
14:27:31 <slacktopus> <hellais> That is what I would suggest. The primary purpose of test-lists is to allow easier sharing of endpoints to use inside of software that does network measurement.
14:28:27 <slacktopus> <khairil.yusof> :thumbsup:
14:28:45 <slacktopus> <khairil.yusof> can we agree to have some sort of explanation on use of the notes field?
14:29:13 <slacktopus> <khairil.yusof> I think guidance on the notes field eg. explanation of website etc. would be good
14:29:37 <slacktopus> <agrabeli> This is mentioned here: https://ooni.torproject.org/get-involved/contribute-test-lists/
14:30:58 <slacktopus> <agrabeli> In general I think the purpose of the Notes column is to describe a website (when useful/necessary to do so), or to add a note about why/how that URL is interesting and therefore should be tracked in the measurements (e.g. it's been blocked before, or it's likely to get blocked because of its content, etc.)
14:31:55 <slacktopus> <khairil.yusof> can we add that to the link you shared? with some examples?
14:32:11 <slacktopus> <agrabeli> @khairil.yusof sure, I'll do that. :slightly_smiling_face:
14:33:10 <slacktopus> <khairil.yusof> sorry I have to look at the lists, and even if it's for the machine readable as source, I'd rather not look up commit history to find out why it was added or what the site is
14:33:20 <slacktopus> <agrabeli> When writing reports, I usually go to the country test list, check the notes, and then compare them with measurements.
14:33:49 <slacktopus> <agrabeli> But I often find that manually going through each blocked website is useful, because it's what allows to tell an interesting story (in ways that a detailed category probably wouldn't).
14:34:27 <slacktopus> <agrabeli> (though it's definitely a tedious process sometimes ,:))
14:35:48 <slacktopus> <khairil.yusof> ok next topic? :slightly_smiling_face:
14:36:01 <slacktopus> <agrabeli> yep
14:36:15 <slacktopus> <agrabeli> topic #2 About Linux users of ooni
14:36:23 <slacktopus> <agrabeli> @kushal would you like to share some words?
14:36:31 <kushal> I am Debian user (in this case), I installed the package from the debian repo (following docs). It can run all the tests, but, can not detect my country or ISP.
14:36:32 <kushal> The latest build/source using docker can detect ISP/country, but, can not run any of the tests. No package in Fedora land. So, please help Linux users by providing
14:36:32 <kushal> something which will work better.
14:38:03 <slacktopus> <hellais> Yeah the debian packaging situation is currently quite a mess
14:38:14 <slacktopus> <hellais> Or even more in general on linux
14:38:15 <slacktopus> <khairil.yusof> I'm running virtualenv + pip right now
14:39:03 <slacktopus> <hellais> I guess the main problem is that we currently don’t have anybody who is actively maintaining or working on the linux packages
14:39:43 <slacktopus> <hellais> I have been kind of ignoring the problem, because my hope is that with the upcoming golang OONI Probe the work of packaging them will be quite improved, but I realise this doesn’t really work in the interim
14:40:23 <slacktopus> <khairil.yusof> I wonder how hard it is to do a snap package
14:40:27 <slacktopus> <sarath_ms> I can pitch in for some Debian packaging work :slightly_smiling_face:
14:40:27 <kushal> i don't have time, otherwise I would have volunteered.
14:41:00 <slacktopus> <sarath_ms> Recently noticed that there is a Failed to Build from Source bug as well.
14:41:18 <slacktopus> <hellais> I am also not a linux user myself, so I can’t really test or check that they are working as they should be working if not in a VM on a one off basis. I can confirm, though, that in many cases it fails and doesn’t work properly.
14:41:21 <kushal> sarathms_, I am using the builds from the ooni site itself.
14:41:44 <slacktopus> <khairil.yusof> happy to test deb packages
14:41:45 <slacktopus> <hellais> Here is a list of known debian related issues with the packages: https://github.com/thetorproject/ooni-probe/issues?q=is%3Aopen+is%3Aissue+label%3Adebian
14:42:18 <slacktopus> <khairil.yusof> for now though virtualenv + pip install is working on Ubuntu 18.04
14:42:22 <slacktopus> <hellais> I started making better debian packages for OONI Probe and they worked inside of the VMs I tried them with: https://github.com/OpenObservatory/ooniprobe-fpm
14:43:05 <slacktopus> <hellais> I think that is the way to go to make packages for the python based ooniprobe (i.e. bundle everything up into a virtualenv and ship all the dependencies inside of a fat debian package)
14:43:50 <slacktopus> <hellais> This is something that is not compliant with the debian packaging policy, but I think we have given up on being included in debian a while ago (and it’s also probably sub-optimal given that our release cycle has been much faster than the debian stable release cycle)
14:44:28 <slacktopus> <sarath_ms> kushal: I haven't tried them yet. I landed on this bug report https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=901537 while looking for something else.
14:44:36 <slacktopus> <darkk> Another part of that problem is that we can't QA all the combinations of libraries (in terms of labor required). E.g. I suspect some incompatibility with latest twisted, but have not debugged the issue yet.
14:44:59 <slacktopus> <sarath_ms> That is when I noticed that the package is currently in the contrib repo. Getting it into the main repo should be a good goal to achieve.
14:45:03 <kushal> darkk, I am guessing so.
14:45:17 <kushal> One option is to pin the Python dependency.
14:45:45 <kushal> Maybe provide a pipfile and a pipfile.lock in the github repo.
14:45:50 <kushal> For something which works.
14:46:12 <slacktopus> <hellais> > That is when I noticed that the package is currently in the contrib repo. Getting it into the main repo should be a good goal to achieve.  I actually think our current goal is to not be inside of debian official repositories, but rather ensure that users will always install OONI Probe with our own package repo.
14:47:09 <slacktopus> <dbyko> :wave:
14:47:13 <slacktopus> <hellais> The reason for that is that old versions of OONI Probe are much less useful to us than the latest ones and it’s very confusing to users to have multiple ways to install OONI Probe on linux and they will most of the times install an outdated “not so useful” version
14:47:56 <slacktopus> <hellais> If somebody is interested in picking up the linux packaging story, I would suggest the following steps:
14:48:02 <kushal> hellais, correct, but, that case we will have to make sure that it also works well :)
14:48:36 <slacktopus> <hellais> 1. Try out the packages built here: https://github.com/OpenObservatory/ooniprobe-fpm and see if they work as expected (they were working the last time I tried them ~ Dec 2017) on the platforms listed in the readme
14:48:53 <slacktopus> <hellais> 2. Rinse and repeat until you get a package that works well in the to be supported platforms
14:49:12 <slacktopus> <hellais> 3. Publish a tested package to bintray and make a call for testing
14:49:43 <kushal> Tonight I will on a Fedora box
14:49:46 <slacktopus> <hellais> 4. Publish the updated and tested package to deb.torproject.org
14:49:46 <kushal> From source
14:50:20 <slacktopus> <hellais> kushal: I also made a test build for fedora in that same repo
14:51:05 <slacktopus> <hellais> It’s not published to bintray though, because it had some issues that I don’t remember (maybe it was related to the system service not using systemd or init?)
14:51:07 <kushal> Ah, I will try that.
14:51:11 <slacktopus> <sarath_ms> > but rather ensure that users will always install OONI Probe with our own package repo.  This is fair.
14:51:46 <slacktopus> <hellais> Ah actually a step 0 to what I outlined above should be:
14:51:46 <slacktopus> <sarath_ms> I was coming from the debian perspective because of projects like FreedomBox https://salsa.debian.org/freedombox-team/plinth/issues/41
14:51:54 <slacktopus> <hellais> 0. Update ooniprobe to 2.3.0
14:52:47 <slacktopus> <hellais> In theory that should just be a matter of bumping the version number in here: https://github.com/OpenObservatory/ooniprobe-fpm/blob/master/build-config.sh
14:53:17 <slacktopus> <agrabeli> @sarath_ms thanks a million for offering to work on the Debian packages, hugely appreciated!
14:53:38 <slacktopus> <agrabeli> In theory we only have 6 more minutes, so let's proceed to the next topic?
14:53:51 <kushal> hellais, we can also pull in the package in Fedora if you want.
14:54:08 <slacktopus> <agrabeli> topic #3 ooni maintained url shorteners
14:54:16 <slacktopus> <agrabeli> Would the person who proposed this like to share some words?
14:54:21 <slacktopus> <tafiti> OONI research involves using and sharing long loooong urls. This exposes them to risk of breaking (one comma and you are out) type of thing and the UI is not very neat, sharing long run.ooni links not the best experience.   Using third party url shorteners opens new problems like easier tracking of testers, for example, using some goo.gl that.   Proposal: Implement an url module for shortening the generated ooni links, explorer urls for footnote
14:54:57 <slacktopus> <hellais> @tafiti we cannot shorten run.ooni links, because the information for what to test MUST be encoded in the URL itself
14:55:40 <slacktopus> <hellais> We can maybe try to use some smarter encoding scheme that is a bit more compact, but we cannot shorten it in the traditional sense of the term
14:56:15 <slacktopus> <tafiti> I do not mean shorten at the running phase but rather at the sharing aspect of it.
14:56:27 <slacktopus> <hellais> If we did make shortened versions of OONI Run links then we would loose the property of having all the information in the URL itself and if, for example, the url shortener website is blocked then it will not work
14:56:33 <slacktopus> <hellais> > I do not mean shorten at the running phase but rather at the sharing aspect of it.  It’s the same.
14:56:35 <slacktopus> <tafiti> I have used google url shortner in the past when I needed to send thrugh the links on IM.
14:56:55 <slacktopus> <tafiti> I hear you.
14:56:56 <slacktopus> <hellais> If you use a google URL shortner the OONI Run link will not work in many edge cases
14:57:14 <slacktopus> <hellais> On iOS (IRC) it will actually never work when passed through a URL shortener
14:57:45 <slacktopus> <hellais> The rules for how mobile deep linking work require that the user taps on a `<a href=` with the target being set to the registered mobile deep link path in the app
14:58:08 <slacktopus> <agrabeli> @tafiti The revamped mobile apps will allow users to test the sites of their choice directly in the app. This doesn't solve the particular problem of sharing URLs to test, but it might be a bit helpful.
14:58:30 <slacktopus> <hellais> If it goes through a chain of redirects before that, it will means that:  a. If the bouncing service is blocked (or there is no connectivity) the app will not open b. In many case it will not even trigger the opening of the app
14:59:06 <slacktopus> <tafiti> I see the pandora with the proposal.
14:59:29 <slacktopus> <hellais> That said, we have heard from various people that they are having difficulties in sharing OONI Run links and we are thinking about how to reduce this friction, but I don’t think that a URL shortener is the solution.
15:00:14 <slacktopus> <hellais> Probably a better way to address this issue is to make some changes to the OONI Run link format to make the information be stored in a more compact way
15:01:47 <slacktopus> <tafiti> My worry had been some platforms may not trigger ooni app when the link is shared but I have never had any of users have that problem. I look forward to testing what you perhaps figure out on ooni.run compactness.
15:02:11 <slacktopus> <t_ooni-slack> Couldn't you run your own link shortener service on a URL that you register with the apps?
15:03:07 <slacktopus> <hellais> @t_ooni-slack the thing is that then we have created another problem: “How do we connect to the URL shortner service (or whatever endpoint resolver $SHORT_URL into $LIST_OF_STUFF to test)”
15:03:20 <slacktopus> <hellais> The beauty of OONI Run links is in a way it’s simplicity
15:03:57 <slacktopus> <hellais> All the information about which URLs to test and how to run the test is encoded in the URL itself, so as long as you are able to deliver the URL, the phone doesn’t even need to have internet connectivity to get the list of sites to test
15:04:16 <slacktopus> <hellais> An example of a OONI Run link is this:  ``` https://run.ooni.io/nettest?tn=web_connectivity&ta=%7B%22urls%22%3A%5B%22http%3A%2F%2Fapple.com%22%5D%7D&mv=1.2.0 ```
15:04:28 <slacktopus> <t_ooni-slack> Sure, things would get more complicated and it is maybe not worth introducing more complexity
15:04:32 <slacktopus> <hellais> This link includes the name of the test to run `web_connectivity` and the list of URLs to test `https://apple.com`
15:04:56 <slacktopus> <hellais> This means that for the OONI Probe app to open and to know which sites to test, it doesn’t need to make *any* network request
15:06:00 <slacktopus> <tafiti> make *any* network request -- This make sense now.
15:06:35 <slacktopus> <tafiti> We are over time - thank you for the perspectives on this.
15:06:51 <slacktopus> <agrabeli> So in summary, we'll try to make the OONI Run links more compact, while brainstorming on other possible solutions.
15:07:00 <slacktopus> <agrabeli> We are indeed overtime
15:07:10 <slacktopus> <agrabeli> Thanks so much everyone for joining us today!
15:07:14 <slacktopus> <hellais> > So in summary, we’ll try to make the OONI Run links more compact, while brainstorming on other possible solutions.  Or rather we will brainstorm other solutions to make OONI Run links more compact :P
15:07:39 <slacktopus> <agrabeli> We'll be announcing the July OONI Community Meeting on the #ooni-talk mailing list (as always): https://lists.torproject.org/cgi-bin/mailman/listinfo/ooni-talk
15:08:03 <slacktopus> <agrabeli> It will most likely be during the last week of July, but if you have suggestions for a date and time that works best for you, please let us know.
15:08:26 <slacktopus> <agrabeli> And if you have further questions or would like to discuss further, the OONI team is available on this channel almost always
15:08:36 <slacktopus> <agrabeli> Thanks everyone! Have a great day/night :slightly_smiling_face:
15:08:44 <slacktopus> <t_ooni-slack> :wave:
16:23:18 <hellais> #endmeeting