13:59:03 <karsten> #startmeeting
13:59:03 <MeetBot> Meeting started Thu Apr 21 13:59:03 2016 UTC.  The chair is karsten. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:59:03 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
13:59:13 <Letty> hi iwakeh, hi karsten
13:59:14 <oma> hi meeting!
13:59:17 <karsten> err, should have said #startmeeting metrics team
13:59:24 <karsten> hi Letty and oma!
13:59:24 <iwakeh> yes :-)
14:00:09 <karsten> https://pad.riseup.net/p/zUNzEIFRq5S4 <- agenda pad
14:00:56 <iwakeh> should I start?
14:01:12 <karsten> ah, let's wait a few minutes for people to add items.
14:01:25 <karsten> we usually start with topics at :05
14:01:40 <iwakeh> ok :-)
14:04:20 <karsten> okay, let's start.
14:04:25 <karsten> * Team documentation wiki page (iwakeh)
14:04:42 <iwakeh> we held a brainstorming session april14th
14:04:42 <virgil_> Hi all
14:04:59 <iwakeh> and the summarized result can be found
14:05:05 <karsten> hi virgil_!
14:05:11 <iwakeh> here https://trac.torproject.org/projects/tor/wiki/org/teams/MetricsTeam/Documentation
14:05:30 <iwakeh> it was also send to the metrics-list
14:05:54 <iwakeh> for also those reading here the invitation to
14:06:03 <iwakeh> also take a look and add questions!
14:06:23 <karsten> thanks for cleaning up the brainstorming notes.
14:06:38 <karsten> I'd also be curious to learn what other questions people might have.
14:06:49 <iwakeh> there was one new
14:06:54 <karsten> yep.
14:06:55 <iwakeh> about tutorials
14:07:58 <karsten> and once we have a complete set of questions, we'll have to start getting answers.
14:08:05 <iwakeh> right, and
14:08:18 <iwakeh> structure the docs we have and create the new ones.
14:08:23 <oma> I took a quick look but the page was a bit overwhelming. Lots of topics. Couldn’t get around figuring out what’s might be missing
14:09:17 <iwakeh> ok, maybe just think about what you would ask
14:09:25 <iwakeh> or have asked in the beginning
14:09:28 <karsten> oma: you could make your own list of questions, from scratch, and then see... heh, yes, that.
14:09:31 <Letty> is there something about the languages included? (used and wanted languages fe)
14:09:43 <karsten> "What skills are required for participating? "
14:09:50 <iwakeh> no problem if something shows up several times, we'll clean it up.
14:09:54 <karsten> could be more explicit about languages. ^ Letty
14:10:17 <oma> "is it okay if I suck at Java/C/…?"
14:10:30 <iwakeh> maybe first the languages used in existing projects?
14:10:32 <Letty> karsten: yes. like (tools, languages…)
14:11:23 <karsten> should we throw it in a pad, so that people can edit after this meeting, and then copy back the result afterwards?
14:11:32 <Letty> that would be great!
14:11:44 <iwakeh> agreed!
14:12:08 <karsten> okay, let me do that. but please wait with the editing until after the meeting, or this will become a second brainstorming meeting.
14:12:16 <Letty> ok
14:12:19 <Letty> :D
14:12:26 <karsten> https://pad.riseup.net/p/dk4mb5lwcSZV
14:13:45 <karsten> ok.
14:13:59 <karsten> should we continue with the next agenda item?
14:14:15 <iwakeh> yes, ctip = CollecTorImprovementProject
14:14:33 <karsten> * CollecTor improvement project (iwakeh)
14:15:09 <iwakeh> I currently work on the build
14:15:27 <iwakeh> environment
14:15:36 <iwakeh> many small steps in the various ctip tickets.
14:15:50 <iwakeh> A main topic is
14:15:51 <karsten> can you tell me the tickets where my response would be most helpful?
14:16:05 <iwakeh> the log analysis
14:16:16 <iwakeh> ticket b/c here
14:16:23 <karsten> I'm trying to respond to all of them, but I need to prioritize this week.
14:16:54 <iwakeh> not here but there is a first reasonable analysis of the missing descriptors from the available logs.
14:17:27 <iwakeh> the other tickets just can be worked on without that much feedback.
14:17:40 <iwakeh> right now. that is :-)
14:17:53 * iwakeh looking up the number
14:17:56 <karsten> okay, I'll look at the missing descriptors analysis soon.
14:18:26 <iwakeh> it's 18798.
14:18:36 <karsten> yep, great.
14:18:45 <iwakeh> it actually doesn't look
14:18:53 <karsten> (if you write ticket numbers as #18798, the bot will mention the link.)
14:19:15 <iwakeh> Oh i know just pasted the wrong thing.
14:19:34 <iwakeh> the page is here https://trac.torproject.org/projects/tor/wiki/doc/CollecTor/AnalysisDescriptorCompleteness
14:20:03 <iwakeh> are there more logs available?
14:20:27 <karsten> more logs as in more recent logs?
14:20:35 <karsten> or more detailed logs?
14:20:45 <iwakeh> or back in time, in case they are there
14:20:56 <iwakeh> no, detail is fine
14:21:09 <karsten> I don't think we have older logs. I could provide more recent ones though.
14:21:23 <iwakeh> yep, that would help as
14:21:24 <karsten> if that's useful, I'll do it later today.
14:21:43 <iwakeh> we currently only have a short 'normal' stretch of time covered
14:22:05 <karsten> ok, sure.
14:22:19 <iwakeh> was there
14:22:33 <iwakeh> an incident on April 1st?
14:23:10 <karsten> [java] java.io.IOException: No space left on device
14:23:15 <karsten> yes.
14:23:24 <iwakeh> ah, thanks.
14:23:38 <virgil> I'm looking for the meeting pad, but it's not https://pad.riseup.net/p/dk4mb5lwcSZV ?
14:23:53 <karsten> https://pad.riseup.net/p/zUNzEIFRq5S4
14:23:54 <Letty> virgil https://pad.riseup.net/p/zUNzEIFRq5S4
14:24:19 <karsten> okay, thanks for making great progress on improving collector, iwakeh!
14:24:41 <karsten> I'll send more logs and look at the graphs later today or tomorrow.
14:24:47 <iwakeh> great!
14:24:52 <karsten> (others, feel free to look, too.)
14:25:07 <iwakeh> tomorrow or next week would
14:25:16 <iwakeh> also be fine and more logs
14:25:47 <iwakeh> our pad is being used
14:26:17 <karsten> you mean our agenda pad? I think that's virgil, right?
14:26:18 <iwakeh> is that on topic there?
14:26:24 <iwakeh> yes
14:26:29 <virgil> yes.
14:26:42 <karsten> okay, let's move on to the next item?
14:27:01 <karsten> * Directory authority disagreement visualization (Letty)
14:27:13 <Letty> after lots of swearing about java/eclipse/everything..the implementation in metrics-web repo is (nearly) done! yay! just need a few infos from karsten.
14:27:20 <Letty> some refactoring is also needed. my code looks like a catastrophe.
14:27:38 <karsten> hehe
14:27:47 <karsten> any quick infos I can give right now?
14:27:58 <karsten> maybe they're useful for others. or hey, maybe others have input, too?
14:28:05 <Letty> maybe
14:28:14 <Letty> file writing, where? same directory as csv?
14:28:20 <iwakeh> if there are java issues, I could help?
14:28:23 <karsten> sure.
14:28:48 <Letty> its not a java issue iwakeh, i quit java programming some years ago :D
14:28:49 <karsten> Letty: in any case, feel free to make such decisions yourself, and if we need to change them later that's really easy.
14:28:54 <Letty> ok
14:28:57 <iwakeh> ok :-)
14:29:09 <Letty> and the other question is: helper classes, where to put them? as private classes in the main file?
14:29:11 <virgil> How is this visualization being done?
14:29:27 <virgil> can you show me an example of the type of graphic?  Because initially it's unclear to me how to visually represent this.
14:29:32 <Letty> virgil: javascript (d3 framework)
14:29:36 <virgil> something like Radar diagrams ?
14:29:53 <virgil> example graphic on non-Tor data?
14:30:00 <karsten> Letty: the whole thing is in a single java file, so yes, maybe keep new classes in that file, unless it gets too big, in which case same package would be fine.
14:30:13 <Letty> virgil: radardiagrams? no one understands them. just visualization ppl like me who think they are awesome
14:30:42 <Letty> karsten: ok thanks. questions answered :)
14:30:45 <virgil> I'm not recommending radar-diagrams, I'm just asking what visualization method you're using.
14:30:50 <virgil> what kind of graphic
14:31:41 <Letty> its like stacked bars with a fix zero point and for every flag a row. wait .. here : http://letty.io/tor/
14:31:41 <karsten> Letty: okay, great! let me know when the code is ready for review, and I'll take a look.
14:32:09 <Letty> karsten: did you see the pm i sended you? about the eclipse issue?
14:32:51 <karsten> ah, I did, and I didn't respond, because I think I wanted to suggest to bring it up here.
14:33:06 <karsten> mind if we bring that issue here?
14:33:54 <nickm> (rehi)
14:34:22 <Letty> ah ok, so i have sometimes trouble with eclipse. first it happend that i couldn't add further functions to the main file.. i just cloned it again and that issue was solved.. but.. i still have problems that ctrl+space (autocomplete) and cmd+click (go to implementation) is not working :/
14:34:50 <karsten> is that for metrics-lib stuff?
14:35:01 <Letty> oh sorry, no its metrics-web
14:35:21 <oma> I use IntelliJ Idea… and I really like it
14:35:22 <karsten> hmm, but isn't that the repository you're working in?
14:35:46 <Letty> jup, it is
14:35:55 <Letty> and in other projects, it worked
14:36:08 <Letty> maybe there are some eclipse bugs.. dont know.
14:36:27 <karsten> do you have an example?
14:36:52 <karsten> (not sure if we can debug this quickly.)
14:37:41 <Letty> syso -> ctrl+space -> should write System.out…..
14:37:51 <karsten> ah. hmmmmm.
14:37:54 <Letty> but it shows nothing
14:38:02 <karsten> it does that here.
14:38:13 <iwakeh> it could also be the eclipse plugins ?
14:38:41 <karsten> maybe copy the dir, delete the .eclipse stuff (and whatever other hidden files there are besides .git),
14:38:46 <karsten> and reimport?
14:39:19 <iwakeh> maybe compare the eclipse plugin versions used to the ones  in karstens setup?
14:39:37 <Letty> ok i will try
14:39:39 <karsten> Version: Luna Service Release 2 (4.4.2)
14:39:51 <Letty> Version: Mars.2 Release (4.5.2)
14:39:53 <iwakeh> oh there is the complete listing
14:40:06 <iwakeh> of plugins somewhere near version or so?
14:40:22 <iwakeh> currently I do not use eclipse anymore
14:40:34 <iwakeh> that's why I'm unspecific here.
14:40:50 <karsten> weird.
14:40:56 <karsten> but let's take this offline.
14:41:05 <Letty> i dont use any plugins .. hrm.. maybe i search for other options. thanks karsten and iwakeh
14:41:08 <Letty> jup
14:41:09 <karsten> it seems unlikely that we'll find out now.
14:41:13 <karsten> great.
14:41:18 <iwakeh> eclipse has these troubles.
14:41:41 <karsten> let's all just use vim. ;)
14:41:47 <karsten> next topic?
14:41:55 <iwakeh> well .. emacs :-)
14:41:59 <Letty> emacs!
14:42:03 <Letty> :D offtopic..
14:42:04 <karsten> uh oh.
14:42:07 <karsten> * Analytics server (oma)
14:42:42 <oma> yep
14:42:45 <oma> I'm working on the new Avro-based converter from CollecTor to Avro, Parquet and JSON.
14:42:56 <oma> Refactoring is done (modularization, fancy patterns, comments removed etc).
14:42:57 <oma> And finally all target format encoders are working :D.
14:43:06 <oma> 5 out of 9 converters are written. That work is tedious but should be done soon.
14:43:07 <oma> I do get some fancy NullPointerExceptions at places where I didn't have them with the
14:43:08 <oma> old version of the code. Not sure why and how much trouble there's hidden.
14:43:33 <karsten> NPE where?
14:43:40 <karsten> in metrics-lib code or in your code?
14:43:45 <oma> For testing it would (still) be helpful to have sample descriptors with all fields filled in. The real ones are often very sparsely populated and can't catch all bugs. And ideally they should be build by someone who knows this stuff well (read: not me)
14:43:52 <oma> in my code
14:44:13 <karsten> ok. maybe wrong assumptions based on incomplete metrics-lib documentation?
14:44:23 <karsten> in which case I'd be glad to have reports.
14:44:51 <oma> Probably, but hard to track down without complete examples (for a Java noob like me)
14:45:20 <karsten> for samples, did you try throwing a monthly tarball from each year into the converter?
14:45:25 <karsten> just to see where it breaks.
14:45:51 <oma> no, so far I’m only testing with a few single descriptors for each type
14:45:53 <karsten> writing descriptors by hand is not really that trivial.
14:46:23 <karsten> then that's my suggestion. maybe take out a dozen descriptors per tarball, but be sure to take tarballs from all years.
14:46:36 <oma> hm, but debugging during mass ingestion of all descriptors is not so much fun either
14:47:05 <karsten> just log the file name where it breaks, then look closer at that descriptor.
14:47:12 <oma> but I can try and see what happens. Maybe all (or most) goes well
14:47:18 <iwakeh> well, and you'll have the stack-trace when it brakes.
14:47:32 <karsten> probably not. that's how I tested metrics-lib: throwing in *all* descriptors since 200x.
14:48:08 <oma> will do. that’s it from me
14:48:14 <karsten> okay, great!
14:48:19 <karsten> * Discontinuing Globe (karsten)
14:48:23 <karsten> this will be quick.
14:48:37 <karsten> I did a quick comparison of atlas and globe requests.
14:49:04 <karsten> https://people.torproject.org/~karsten/volatile/atlas-globe-2016-04-01.png
14:49:41 <karsten> both are performing a very similar service, and neither of them is actively maintained in the sense of having somebody who is enthusiastic about adding new features.
14:50:11 <karsten> I want to retire globe by removing it from globe.torproject.org and archiving (?) the globe trac tickets.
14:50:17 <iwakeh> how old are the code-bases? and what?
14:50:47 <karsten> https://gitweb.torproject.org/globe.git/
14:50:49 <iwakeh> what technology? to be more precise .
14:50:53 <karsten> javascript.
14:51:00 <karsten> or more detailed? :)
14:51:05 <karsten> some javascript frameworks.
14:51:11 <virgil> I support decommissioning globe.  Or just leaving it forever as a zombie (if it takes no effort to maintain)
14:51:26 <iwakeh> better take it down.
14:51:31 <karsten> there's also a zombie version of globe hosted by the author of globe.
14:51:48 <karsten> which we'd not take offline, of course.
14:51:58 <iwakeh> then it is still running there that'll be fine.
14:52:01 <karsten> I think it's bad to keep things around as zombies.
14:52:09 <iwakeh> agreed.
14:52:15 <karsten> because people expect us to maintain things hosted on .torproject.org.
14:52:44 <Letty> uhh.. first class i opened looks more like java.. not like javascript… intressting..
14:52:57 <karsten> okay, so my plan is to announce this on tor-project@ soon.
14:53:10 <karsten> which might produce some "oh-no, please don't" responses.
14:53:35 <iwakeh> well, point at the other instance?
14:53:46 <karsten> but anyway, I mostly wanted to share my plan here. if you have any input, please let me know (via email, running out of time here).
14:53:47 <virgil> Karsten, as the administrator of "metrics", you should have final say.  It's not a consensus.
14:53:50 <karsten> iwakeh: good point.
14:54:10 <karsten> virgil: well, I prefer consensus anyway.
14:54:19 <karsten> but taking things offline is always hard.
14:54:29 <karsten> anyway, let's move on.
14:54:37 <karsten> * Roster update
14:54:49 <karsten> (is phw even around?)
14:54:58 <karsten> err, Roster update is virgil.
14:55:07 <phw> yes, i'm around.
14:55:14 <karsten> ah, great.
14:55:36 <karsten> virgil: want to give a quick update on Roster?
14:55:57 <virgil> We did not one, but TWO heretical things on Roster.
14:56:13 <virgil> (1) we gave UUIDs to MyFamilys.
14:56:15 <karsten> phw: want to prepare a few sentences what you mean there, to paste them at :59? we might have to postpone discussion until the next meeting...
14:56:28 <virgil> (2) We're doing persistent storage of these UUIDs.
14:56:40 <virgil> we have to do this to be able to trace family contributions back through time.
14:56:49 <karsten> virgil: any chance that we can move that to onionoo?
14:57:01 <karsten> virgil: so that other onionoo clients could use it?
14:57:19 <virgil> karsten: Totally in favor of that.  Do others object?
14:57:36 <karsten> well, can you create a ticket and describe the design?
14:57:51 <virgil> karsten: note that determining family UUID membership is not entirely certain.  Sometimes we guess.
14:58:14 <karsten> yes, I can imagine that it's non-trivial.
14:58:33 <karsten> (and we're running out of time very soon. we need to move this to the next meeting.)
14:58:38 <karsten> really sorry.
14:58:39 <virgil> I wanted to use a method from biology to reconstruct lineages.  But I was putting too much work into it.
14:58:45 <virgil> k, go on.
14:58:52 <karsten> you open a ticket for this?
14:58:57 <virgil> we will
14:59:00 <karsten> thanks.
14:59:06 <karsten> phw: want to paste your thing?
14:59:09 <phw> i did.
14:59:15 <karsten> ah, here.
14:59:20 <phw> using exitmap and an authoritative DNS server, we can identify and analyse the resolvers of exit relays.  some details are online: https://nymity.ch/dns-traffic-correlation/
14:59:25 <phw> so far, we have analysed these resolvers' query quality: 0x20 encoding, DNSSEC support, and random source ports.
14:59:31 <phw> if anyone is interested in measuring other aspects, let me know.
14:59:42 <karsten> cool!
14:59:52 <phw> no need for discussion anyway.  just dumping this here.
15:00:03 <phw> and there we go, finishing at 11:00.
15:00:18 <karsten> we can put it on the next agenda.
15:00:21 <karsten> in two weeks.
15:00:30 <karsten> to the top even.
15:00:37 <virgil> phw---what DNS thing concerns you?
15:00:41 <karsten> same with roster.
15:00:57 <karsten> okay, everyone, thanks for having a very interesting meeting today!
15:01:01 <phw> virgil: whatever allows adversaries to interfere with the resolution process.
15:01:15 <virgil> more specific.
15:01:18 <virgil> forward or reverse DNS?
15:01:31 * karsten ends this meeting, but just ignore that if you want to discuss more.
15:01:41 <karsten> oh, and please add more questions here: https://pad.riseup.net/p/dk4mb5lwcSZV
15:01:44 <iwakeh> that sound interesting. but i have to go. next meeting. bye, bye.
15:01:48 <karsten> #endmeeting