14:00:01 <karsten> #startmeeting metrics team
14:00:01 <MeetBot> Meeting started Thu Sep 22 14:00:01 2016 UTC.  The chair is karsten. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:01 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
14:00:05 <karsten> https://pad.riseup.net/p/3M7VyrTVgjlF <- agenda pad
14:00:16 <karsten> let's collect agenda items there.
14:02:15 <iwakeh> I'm done.
14:02:16 <karsten> anything else?
14:02:19 <karsten> ah ok.
14:02:24 <karsten> guess we can start then.
14:02:28 <iwakeh> right
14:02:29 <karsten> * CollecTor data distribution (iwakeh)
14:02:49 <iwakeh> well, I'm happily coding and testing on the way and
14:03:03 <iwakeh> reading a lot of code from collector and metrics-lib to
14:03:13 <iwakeh> avoid re-inventing stuff.
14:03:17 <iwakeh> But, I also
14:03:31 <iwakeh> see many areas for future improvements.
14:03:55 <iwakeh> Should the
14:04:05 <iwakeh> vote processing be limited to
14:04:21 <iwakeh> the authoroties the receiving collector instance
14:04:28 <iwakeh> contacts?
14:04:51 <iwakeh> "the authorities"
14:05:05 <karsten> what do you mean? where else would the collector instance receive a vote from?
14:05:20 <iwakeh> during sync.
14:05:21 <karsten> ah.
14:05:36 <iwakeh> one kind of spam to avoid.
14:05:43 <iwakeh> easily avoided.
14:06:00 <karsten> well, maybe,
14:06:20 <karsten> but also yet another source of error, when the operator forgets to update their config.
14:06:36 <karsten> I'd say it's okay to accept those votes for now.
14:06:41 <iwakeh> in what way?
14:06:49 <iwakeh> source of error?
14:06:58 <iwakeh> if i collect votes from
14:07:05 <iwakeh> many outdated
14:07:23 <iwakeh> auths that is my business.
14:07:24 <karsten> when a new authority gets added, we'll likely learn that from the consensus.
14:07:32 <karsten> without configuring something.
14:07:41 <karsten> and we should fetch those new votes, too.
14:07:47 <karsten> it would be sad to miss them.
14:08:00 <iwakeh> ok, that would work in general.
14:08:12 <iwakeh> actually
14:08:22 <iwakeh> also for normal collecting?
14:08:36 <karsten> I think that's how normal collecting works.
14:08:49 <karsten> but let me check.
14:08:52 <iwakeh> we use the configured auths.
14:10:20 <karsten> https://gitweb.torproject.org/collector.git/tree/src/main/java/org/torproject/collector/relaydescs/RelayDescriptorDownloader.java#n486
14:10:35 <karsten> we use the configured auths, and we learn about votes we should fetch from the consensus.
14:11:03 <karsten> we do both to be sure we're not missing a vote only because that vote didn't make it into the current consensus.
14:11:07 <iwakeh> but were do we fetch these votes from?
14:11:20 <karsten> from any of the authorities, at random
14:11:21 <karsten> .
14:11:28 <iwakeh> the configured ones.
14:11:37 <karsten> those, too.
14:12:05 * karsten operates from memory here
14:12:29 <iwakeh> ok, i don't think we add urls during download.
14:12:46 <karsten> oh, we do.
14:12:58 <karsten> otherwise we wouldn't know which server descriptors to download, for example.
14:13:12 <iwakeh> hosts, too?
14:13:30 <karsten> no, I don't think we're adding hosts to download from.
14:13:41 <karsten> but we don't have to. in theory, all authorities should have votes from all other authorities.
14:13:45 <iwakeh> that's what I was referring to.
14:14:03 <iwakeh> add hosts=auth host for collection.
14:14:54 <karsten> so, the set of hosts we're downloading from is fixed.
14:15:01 <karsten> but the votes we download is not.
14:15:12 <iwakeh> yes.
14:15:15 <karsten> we can learn about a vote from the consensus, even if we didn't have that one configured.
14:15:29 <karsten> and we can have a vote configured that is currently not referenced from the consensus.
14:15:36 <iwakeh> and when learning about a new auth?
14:15:46 <karsten> we fetch their vote, but not from them.
14:15:53 <iwakeh> that could also be used for download eventually.
14:16:01 <karsten> ah!
14:16:05 <karsten> sure, we could do that.
14:16:06 <iwakeh> and avoid fetching from past auths.
14:16:11 <karsten> yep.
14:16:21 <karsten> plenty of ways to make the downloader even smarter.
14:16:46 <iwakeh> sure, but first the plain new functionality.
14:16:50 <iwakeh> :-)
14:16:54 <karsten> but I understood the question as: what to do while syncing with a vote we didn't expect.
14:17:07 <iwakeh> well, with
14:17:18 <karsten> yes, and we might not have the time to make the downloading smarter in the next 9 months..
14:17:25 <iwakeh> the current design we would sync it.
14:17:29 <karsten> even though it's tempting..
14:17:40 <karsten> yes, and I'd say that is fine.
14:17:51 <iwakeh> That's why I
14:18:08 <iwakeh> suggested using simple criterium classes.
14:18:18 <iwakeh> that facilitates easy selection.
14:18:52 <iwakeh> I'll keep it in mind and see how nuch coding it'll require.
14:19:19 <karsten> hmm, I'm not sure if we want to make it more complex soon.
14:19:34 <iwakeh> won't be more complex, I hope.
14:19:36 <karsten> maybe keep that as a note somewhere for now, rather than in code?
14:19:46 <karsten> ok.
14:20:28 <iwakeh> ok.
14:20:42 <karsten> what's your plan for having something running?
14:20:50 <karsten> and for me to review it?
14:20:57 <iwakeh> September.
14:21:00 <iwakeh> 2016
14:21:02 <iwakeh> :-)
14:21:03 <karsten> hah
14:21:06 <karsten> neat!
14:21:10 <iwakeh> next week.
14:21:17 <karsten> sounds great!
14:21:37 <iwakeh> I test a lot on the way.
14:22:24 <iwakeh> That's all for collector sync.
14:22:31 <karsten> okay. :)
14:22:40 <karsten> * Roadmap for next 6--9 months (karsten)
14:22:48 <karsten> let's make a roadmap.
14:22:53 <iwakeh> where?
14:23:01 <karsten> agenda pad?
14:23:08 <iwakeh> sure.
14:23:40 <karsten> there, I pasted what I already have.
14:23:59 <karsten> want to go through them and add thoughts?
14:24:07 <iwakeh> yes.
14:24:38 <karsten> these are just sponsor X tasks, by the way.
14:24:51 <karsten> we could add more tasks.
14:24:52 <iwakeh> should others be added?
14:25:01 <karsten> if we want to get them done, yes.
14:25:18 <karsten> for example, things that help us with the next grant.
14:26:35 <iwakeh> yes, shiny things ;-)
14:27:00 <iwakeh> or derived from that experience
14:27:03 <karsten> shiny things are sort of included in the Metrics usability analysis.
14:27:56 <karsten> to some extent the choice of shiny over other frameworks (or writing our own thing) influences usability.
14:28:23 <karsten> requires javascript vs. runs in tor browser high security setting.
14:29:07 <karsten> btw, new prototype:
14:29:16 <karsten> https://tor-metrics.shinyapps.io/webstats2/
14:29:51 <iwakeh> takes a long time to load ...
14:30:05 <karsten> how long?
14:30:40 <karsten> I thought that had improved with reducing the detail from daily to monthly.
14:30:59 <iwakeh> in TorBrowser hmm, 30sec.
14:31:28 <iwakeh> now it responds quickly :-)
14:31:34 <karsten> in theory, most of the work should happen on the server.
14:32:09 <iwakeh> This is really nice!
14:32:18 <karsten> glad you like it. :)
14:32:42 <karsten> the cool thing is that it's really easy to write.
14:32:57 <karsten> but! I didn't want to distract you from the roadmap thing. ;)
14:33:11 <iwakeh> well, should we write the usability analysis?
14:33:29 <karsten> you mean we vs. somebody else?
14:33:52 <iwakeh> or, have the prototype compared to the existing ;-)
14:35:37 <karsten> well, I found it useful to build this prototype to learn what we can do with shiny. but I can see the value in an analysis document vs. deciding on another tool and switching directly.
14:36:37 <karsten> another option we have is d3.js, which is supported by shiny, too, I think.
14:36:47 <karsten> I mean, we have lots of options.
14:37:01 <iwakeh> the problem will be maintaining the server installation ...
14:37:12 <karsten> the shiny server?
14:37:15 <iwakeh> not the implmentation of services.
14:37:19 <iwakeh> yes.
14:37:51 <karsten> could be. therefore we save effort for maintaining our own shiny-like code.
14:38:20 <karsten> lots of things to consider. I could imagine that writing this analysis will be fun.
14:38:25 <karsten> and take a while. ;)
14:48:00 <karsten> I wonder if we should shut down the MeetBot.
14:48:06 <karsten> and continue on the pad.
14:48:14 <iwakeh> sure.
14:48:22 <karsten> and by shut down I don't mean... ok.
14:48:23 <karsten> #endmeeting