15:01:00 <asn> #startmeeting prop259
15:01:00 <MeetBot> Meeting started Wed Mar 23 15:01:00 2016 UTC.  The chair is asn. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:00 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
15:01:03 <tdruiva> what is better
15:01:04 <asn> hello there :)
15:01:06 <iapazmino> hi
15:01:08 <asn> ok
15:01:13 <nickm> (meetbot is logging for posterity)
15:01:32 <chelsea_komlo> very handy!
15:01:34 <asn> ok so summary here is
15:01:42 <asn> that the prop259 spec has kind of stabilized
15:01:48 <asn> and now the STRIKE team is implementing it
15:01:56 <asn> to see how it works in practice
15:01:59 <asn> the latest spec is
15:02:00 <asn> :
15:02:06 <asn> https://github.com/twstrike/torspec/blob/review/proposals/259-guard-selection.txt
15:02:13 <olabini> hey all, I'm here
15:02:17 <asn> #action we should push the latest version of the spec on [tor-dev]
15:02:27 <asn> olabini: greetings
15:02:34 <asn> ok you guys want to brief us up how the implementation is going?
15:02:51 <asn> or you want to start off in a different way?
15:03:23 <olabini> rjunior: you want to lead this?
15:03:34 <rjunior> sure
15:04:00 <rjunior> we have implemented the proposal in C
15:04:08 <rjunior> it's in this branch
15:04:09 <rjunior> https://github.com/twstrike/tor_for_patching/tree/prop259
15:04:32 <rjunior> I was more interested in finding the right place to plug things into tor
15:04:33 <nickm> is there a guardsim branch that matches the proposal and matches the C implementation?
15:04:40 <rjunior> yep
15:04:45 <rjunior> https://github.com/twstrike/tor_guardsim/tree/develop
15:05:00 <asn> i still have not seen the graphs from this simulation.
15:05:06 <rjunior> and I also have some graphs for the simulation
15:05:15 <rjunior> in a google doc here: https://docs.google.com/document/d/1meFgonpSzz_FEa2LMSRyt6AOaEcq27PCZSyEi7kqZeE/edit
15:05:35 <rjunior> back to the C code
15:05:57 <rjunior> the strategy has been using as much as possible of the existing tor code
15:06:13 <rjunior> so we dont face any problem extern to the proposal
15:06:44 <rjunior> in summary, we replaced choose_random_entry_impl by the proposal
15:06:58 <asn> aha
15:07:30 <rjunior> trying to roughly keep its semantics (adding chosen guards to the global "entry_guards", for example)
15:07:52 <asn> (The travel network case is not in the graphs, eh?)
15:08:20 <rjunior> and plugging everything else needed to dont break tor (neither because of the proposal, or code outside of it)
15:09:04 <rjunior> we still need to decide if this is the better place to put the proposal
15:09:34 <asn> which place?
15:09:47 <rjunior> and I'm currently interested in finding existing behavior that might be important but we are ignoring due the way we plugged the proposal
15:10:05 <rjunior> replacing choose_random_entry_impl
15:10:32 <rjunior> or if we shoudl keep it, and replace more granular functions (like choose_random_entry and router_pick_directory_server)
15:10:38 <asn> well, you will definitely need to replace choose_random_entry_impl() to implement this proposal. you will probably have to touch other places as well.
15:10:47 <rjunior> choose_random_entry_impl seems to be what we want
15:10:52 <asn> i agree
15:10:58 <asn> so does the thing work now?
15:11:02 <fanjiang> https://github.com/twstrike/tor_for_patching/issues
15:11:11 * asn looks
15:11:16 <nickm> I think it might be a good idea to step back.
15:11:25 <nickm> Which do we actually have more confidence in now, the python, or the proposal?
15:11:28 <rjunior> I work (it does not crash)
15:11:57 <nickm> Not having read either too closely in a while, I am more confident that the Python describes something actually buildable in Tor, in sufficient detail.
15:12:11 <nickm> I'm less sure that the proposal really specifies a complete algorithm, rather than just describing it.
15:12:25 <nickm> But I don't actually know.  What's the status on those?
15:13:08 <rjunior> the python implementation and the proposal are in sync
15:13:20 <rjunior> and the C implementation is in sync with the proposal too
15:13:29 <nickm> I'm not asking that part.
15:13:34 <nickm> I'm asking about the level of detail.
15:13:43 <rjunior> I'm not sure what is missing to make the proposal really specify a complete algorithm, rather than just describing it
15:13:47 <fanjiang> Though, there are some details in C we need to confirm yet
15:14:02 <rjunior> exactly.
15:14:03 <nickm> If three people sat down with the proposal, would they all come up with a python implementation that chose guards in the same way?
15:15:19 <iapazmino> they should, assumptions have been added to the spec
15:15:45 <iapazmino> nevertheless, some concepts in the spec might not be exactly equivalent to those in the tor code
15:16:10 <iapazmino> these are the details which need confirmation fan was talking about
15:16:15 <rjunior> we have ignored details on the current implementation and focused on specifying a strategy for choosing guards. things like how specific tor configuration properties should change the strategy, how the guard selection integrates with other parts beyond building a circuit
15:16:40 <asn> ok
15:16:40 <rjunior> and this is where we are
15:16:51 <asn> nickm: any questions about the spec? any specific doubts?
15:17:13 <asn> otherwise maybe we can go into the spec / code questions.
15:17:24 <asn> that rjunior has
15:18:01 <nickm> I guess I don't see how it corresponds to the "entry points" we see in the guardsim code.  I would expect to see part of it written in terms of "When a router goes down...."  and "When we're building a circuit...." and "When Tor starts for the first time..."
15:18:20 <nickm> but maybe I need to read more closely
15:18:29 <nickm> in any case, let's move on to the spec/code questions?
15:18:43 <rjunior> so in reply to nickm I believe no, the spec lacks details needed to make sure 3 people will implement the same thing
15:19:35 <rjunior> ok
15:19:54 <asn> rjunior: hm, what are these details?
15:20:52 <chelsea_komlo> or detail about where it fits into the entire action of connecting, rather than just specifying  it in isolation
15:21:12 <rjunior> how to behave when state->need_uptime or state->need_capacity
15:21:25 <rjunior> is one specific example
15:21:38 <fanjiang> For now, some missing part of spec/python according to C are some detailed options and boundary cases, as mentioned. Some are the unspecified entry points.
15:21:58 <rjunior> or how ReachableAddresses should influence the guards we choose
15:22:45 <rjunior> and every implementation detail we might find while reading choose_random_entry_impl
15:22:54 <asn> chelsea_komlo: indeed, i don't think the proposal currently specifies really well how it fits into the Tor networking system,
15:23:13 <asn> there is the pseudocode in the appendix, but that does not reflect the tor reality as close as we would like
15:23:18 <asn> (as discussed in the recent mailing list thread)
15:23:18 <rjunior> I think we should document every aspect of choose_random_entry_impl
15:23:37 <chelsea_komlo> great, that can be an action item
15:23:48 <asn> i was expecting that as you guys get more familiar with how tor works, the pseudocode in the end would change slightly.
15:24:07 <rjunior> so the idea is making the proposal more implementation oriented (giving the current tor networking system) rather than more descriptive
15:24:10 <nickm> I would be happy for somebody (possibly with tor experience) to also take on making Tor's internals look more like the guardsim api
15:24:32 <nickm> given that the guardsim api is actually almost sane, but the current internals that tor has are pretty messed up
15:24:48 <asn> nickm: indeed that would be nice
15:25:13 <nickm> that'll need a careful look at the changes, though, and a little planning
15:25:20 <asn> #action figure out how exactly the prop259 algorithm fits into tor
15:25:23 <asn> there. an action time.
15:25:24 <asn> item.
15:25:33 <nickm> [also I'm in two meetings at once now so please poke me again if there's a question i'm not answering]
15:25:39 <asn> <rjunior> how to behave when state->need_uptime or state->need_capacity
15:26:01 <asn> that's also a good point. the current proposal does not specify what happens when we need a guard for a circuit with restricted requirements.
15:26:28 <asn> like what happens if your top guard is not fast, and you need a fast circuit.
15:27:13 <asn> the current tor algorithm just walks through all the guards from the top, and chooses the next guard that is fast on your list.
15:27:38 <asn> do you think that's wrong behavior? or just unspecified?
15:27:54 <nickm> armadev: am I wrong, or are all guards not fast-enough and stable-enough?
15:28:07 <nickm> The guard requirements right now include speed and stability
15:28:08 <asn> there are some very few guards that are not fast.
15:28:13 <asn> or there used to be.
15:28:16 <rjunior> this is the kind of details we didnt not sure exist until we started to implement the C code
15:28:27 <asn> rjunior: doesn't surprise me, yes.
15:28:37 <nickm> I think that using any guard for those is okay.  but I'd like armadev's opinion too
15:29:10 <rjunior> so i created an issue to create a list of this specific things and discuss them with you: https://github.com/twstrike/tor_for_patching/issues/3
15:29:33 <rjunior> for now, need_uptime and need_capacity are the ones I remember
15:29:34 <rjunior> and aldo
15:29:49 <rjunior> how should ReachableAddresses influence the guard selection
15:29:59 <armadev> right, i think everything with the Guard flag meets some pretty good standards for speed and being-there-often
15:30:16 <armadev> (if that was the question)
15:30:17 <fanjiang> At least, sampled guards may need some re-definition
15:30:34 <rjunior> ah, and also entry guards vs directory guards
15:30:57 <asn> hm i'm not sure how exactly the current guard algorithm works with ReachableAddresses.
15:31:37 <asn> rjunior: what is the uqestion about directory guards?
15:31:50 <nickm> I believe that we shouldn't connect to any guard not included in ReachableAddresses.  If we've picked one already, we shouldn't throw it away, but we shouldn't use it.
15:31:55 <rjunior> when creating the sample sets, we assume it is very likely that the sample will contains directory guards
15:31:58 <nickm> (not sure whether this is the current behavior or not)
15:32:31 <rjunior> otherwise we would need to create additional sample sets (for directory guards)
15:32:34 <asn> rjunior: indeed it is very likely, since most guards are also directory guards.
15:33:13 <rjunior> great
15:33:14 <asn> rjunior: has this been a problem to you?
15:33:34 <asn> like, that you populate your sampeld guards list, and then there is a circuit that asks for a directory guard and you don't have one?
15:33:55 <rjunior> no because we have skipped the directory guards until yesterday =D
15:34:06 <isis> hello and sorry i'm late
15:34:13 <asn> rjunior: OK, I see.
15:34:13 <nickm> hi isis
15:34:30 <iapazmino> asn: should we then use the fallback directories listed in fallback_dirs.inc?
15:34:40 <asn> iapazmino: how would you use them?
15:34:41 <tdruiva> hi isis
15:34:47 <rjunior> but this is the current strategy, assuming it's very likely we will find enough directory guards in the sample sets
15:35:08 <asn> rjunior: indeed.
15:35:12 <asn> rjunior: although the way the algo currently works
15:35:23 <asn> if the algo is called first with dir=True
15:35:33 <asn> then the lists will be populated only with dirguards
15:35:36 <asn> no?
15:35:46 <rjunior> this is my understanding
15:36:00 <iapazmino> asn: I thought these were the routers to be used when nothing else was left
15:36:09 <rjunior> but then, the next time you call with Dir=False, this guard will be ignored
15:36:14 <iapazmino> asn: did not go that far as to think on how to use them
15:36:25 <rjunior> unless there is no other live entry guard
15:36:33 <asn> rjunior: why?
15:36:38 <asn> all directory guards can be normal guards.
15:36:44 <rjunior> oh
15:36:47 <rjunior> I see
15:36:57 <rjunior> in this case, yes this will be used until it fails
15:37:04 <asn> directory guards are guards that can also answer directory requests.
15:37:14 <nickm> we're moving towards a world where most relays are directory caches by default.
15:37:17 <asn> in the future all guards will be able to answer directory guards, and hence all guards will be directory guards.
15:37:39 <asn> currently about 80% of guards are directory guards. so there is a small chance your top guard will not be and you will need to move to the next one.
15:38:14 <rjunior> so in the future there should be no distinctions between directory and entry guards
15:38:36 <asn> yes. in an ideal world, your main guard will also be answering your directory queries.
15:38:58 <rjunior> ack
15:39:03 <rjunior> about ReachableAddresses, we have decided to only use it to determine the sample set of dystopic guards
15:39:41 <asn> plausible.
15:39:46 <rjunior> the idea being: a node is dystopic if it matches the ReachableAddresses (or if its port is 80/443 in the absence of ReachableAddresses9
15:39:53 <asn> plausible yes.
15:40:19 <rjunior> but according to nickm we shouldn't connect to any guard not included in ReachableAddresses
15:40:38 <asn> indeed, so if you have already added a guard and then ReachableAddresses changed...
15:41:11 <asn> and the new ReachableAddresses does not include that guard, you should not try to connect to it.
15:41:27 <rjunior> meaning that the utopic sample set should also obey ReachableAddresses, correct?
15:42:14 <asn> i guess so. but i'm not sure if this restriction should be performed on the "pick the guard" layer, or on the "connect to the guard" layer.
15:42:20 <asn> not sure where it happens now either.
15:42:27 <rjunior> otherwise we would try to connect to guards not included in ReachableAddresses
15:42:30 <rjunior> ack
15:42:42 <Sebastian> asn: I just got this on my 0.2.7.6 hs
15:42:53 <rjunior> so we stil pick the guard, but dont connect to it
15:42:54 <Sebastian> Introduction circuit established without a rend_intro_point_t object for service
15:43:01 <rjunior> and why making such distinction is important?
15:43:02 <Sebastian> Isn't that supposed to be fixed?
15:43:31 <armadev> asn: if you come up with a (hopefully short) clear list of "how does tor do X now" or "what function should I look at to understand Y", i'll try to be helpful.
15:43:38 <Sebastian> sorry for interrupting, /me waits
15:43:41 <armadev> re guards, entry node selection, etc
15:43:48 <isis> if it happened at the "connect to the guard" layer, then the connetion wouldn't happen, that layer would ask for a guard from the "pick a guard" layer, which would not understand why
15:44:03 <armadev> asn: i confess i haven't read the (hidden, not sent to tor-dev) proposal that is being discussed here
15:44:24 <asn> everything has been happening on tor-dev, it's just that we didn't send an individual mail about the proposal.
15:44:32 <asn> armadev: but yes, I will take your offer on this.
15:44:49 <asn> armadev: next time rjunior et al. ask me about some functionality i don't know about.
15:45:09 <armadev> great. expect high latency on the responses though (maybe this helps you pare down the list)
15:45:37 <asn> bleh sorry my mind drifted. where were we
15:45:58 <asn> rjunior: hmmm
15:46:10 <asn> rjunior: i'll need to look at how REachableAddresses works in tor right now to answer your question better.
15:46:39 <asn> rjunior: if you compile a list of questions after this meeting, I will answer them in the next two days.
15:46:59 <rjunior> asn: this is my plan for the next 2 days
15:47:10 <asn> rjunior: btw, did you find all the places to hook your code?
15:47:28 <asn> i remember you were puzzled about the "while True" loop and stuff on the mailing list.
15:47:36 <rjunior> everything else I've sent in last week email, we have find a working solution
15:47:51 <asn> oh wow that's nice
15:47:53 <rjunior> I didnt have to to anything about it
15:48:00 <fanjiang> while true is a goto hah
15:48:03 <asn> rjunior: yes doesn't surprise me.
15:48:13 <rjunior> the current way tor keeps trying to build a circuit until it success, matched with that loop
15:48:20 <asn> rjunior: because the execution will just come back to you; you don't need to make the loop yourself.
15:48:38 <asn> ok
15:48:45 <rjunior> I assumed it to be an implicit rather than explicit loop, as your email implied
15:48:50 <asn> ack
15:49:02 <asn> i'm curious to see how you have fitted the proposal in tor.
15:49:06 <asn> do you think i should look at the code?
15:49:13 <rjunior> you can
15:49:31 <asn> #action finally look at the code
15:49:38 <asn> #action answer rjunior's implementation questions
15:49:45 <rjunior> if you configure with --enable-prop259
15:49:48 <asn> alright
15:49:52 <rjunior> you should be able to run it
15:49:53 <asn> interesting
15:49:58 <asn> does it have any nice debug logs?
15:50:12 <asn> that i can monitor my guard usage with?
15:50:13 <rjunior> not many, this is a good idea
15:50:28 <asn> yes please
15:50:32 <rjunior> to be more detailed in debug logging, for the moment at least
15:50:37 <chelsea_komlo> that could be an action item as well
15:50:59 <asn> #action write beautiful & intuitive debug logging to monitor guard usage
15:51:20 <asn> my plan for the future is to run the proposal on my laptop for a month or so and manually monitor the guard usage, so good logs will really help.
15:51:57 <rjunior> last question from my side
15:51:59 <asn> so like where did you put the SHOULD_CONTINUE thing?
15:52:12 <rjunior> nickm mentioned some work on the modularity documentation during the network team meeting
15:52:13 <asn> in which tor function?
15:53:45 <rjunior> I assumed it to be the same thing as how channel_do_open_actions() uses entry_guard_register_connect_status()
15:54:08 <asn> ack
15:54:21 <rjunior> if entry_guard_register_connect_status() returns false in this function, it drops the chosen entry_guard and "continue"
15:55:05 <asn> rjunior: ah nice.
15:55:06 <rjunior> because should continue exists to simulate this existing behaviour, afaik
15:55:07 <asn> got it.
15:55:35 <asn> btw, wrt implementation details like reachable addresses, i would suggest you do the same thing that the current tor algorithm is doing when possible
15:55:52 <asn> if it's blocking you from progress
15:55:55 <rjunior> so, yeah, the integration is spread over different places due how the tor networking code currently is
15:56:05 <asn> but then also note it down somewhere so that we can look deeper into it.
15:56:08 <asn> rjunior: right.
15:56:29 <rjunior> back to mmy last question, Is there something we can start taking into consideration for prop259 in the subject of modularity?
15:56:44 <asn> nickm: ^
15:57:14 <asn> rjunior: obvious things is to make a new file for the proposal and hide in there as much functionalityu as possible
15:57:18 <rjunior> asn: you mentioned a "Updated draft: improved modularity in tor" during the tor networking team meeting
15:57:27 <asn> ideally your interface will be much more elegant than the current entrynodes.c .
15:57:30 <rjunior> but I suspect it is not public (I couldnt google it)
15:57:58 <asn> nickm: can we send your modularity doc to rjunior ?
15:58:07 <asn> rjunior: i think nickm is going to push it to the mailing list RSN
15:58:12 <rjunior> nice
15:58:24 <rjunior> no more questions from me
15:58:27 <asn> rjunior: also, *not* using the global entry_guards list would be nice.
15:58:43 <nickm> yes, send to rjunior .  Also, I plan to send today.
15:58:50 <asn> ack thx
15:59:11 <asn> rjunior: ideally the whole guard algorithm thing could be objectified. and then we could have multiple of them running in parallel.
15:59:27 <asn> rjunior: in the sense, that we could have guard layers (prop247), and each layer needs a different guardlist.
15:59:32 <asn> rjunior: you can't do that with global structures.
15:59:59 <asn> #action send modularity doc to rjunior and team
16:00:54 <rjunior> any other question from the team?
16:01:01 <asn> rjunior: also, what are you currently doing about needs_capacity, etc.?
16:01:09 <rjunior> or any other question / action item from the tor team to us?
16:01:17 <iapazmino> just one around the metrics on the simulation
16:01:18 <rjunior> we are just ignoring
16:01:23 <tdruiva> the ones that I have can be answered by email ;)
16:01:30 <asn> tdruiva: great :)
16:01:35 <asn> iapazmino: go for it
16:01:47 <asn> another question: how much time you have left on this task?
16:01:58 <iapazmino> should we reconsider the numbers we are gathering in different scenarios?
16:02:06 <tdruiva> we're planing one week momre
16:02:14 <tdruiva> *more
16:02:27 <asn> tdruiva: and then you are gone for real. fun.
16:02:41 <asn> iapazmino: what do you mean?
16:02:45 <iapazmino> i mean, time and successful connections are not equally good metrics on a stable network as in a network-down scenario
16:02:49 <tdruiva> \o/ well, we're working for it
16:03:06 <rjunior> asn: I thinks it' more like "and then we are not full time on this"
16:03:21 <asn> ok
16:03:22 <tdruiva> also, how do you wanna to test before to get live?
16:03:23 <olabini> asn: we won't be gone
16:03:33 <olabini> asn: we are committed to shepherding this to completion
16:03:41 <olabini> asn: but our hope is that the bulk of the work will be done by then.
16:03:44 <asn> olabini: ack
16:03:57 <asn> iapazmino: what do you suggest instead?
16:04:15 <asn> tdruiva: what do you mean "before to get live"?
16:04:19 <tdruiva> olabini, +1
16:04:26 <asn> tdruiva: before you push the patch for review?
16:04:47 <iapazmino> asn: i guess this would be a per-scenario thing, depending in what success means to each
16:04:58 <asn> [another question: i don't see the traveling network on the graphs list. should i care?]
16:05:01 <tdruiva> asn, yes
16:05:07 <asn> tdruiva: hmmm
16:05:14 <asn> tdruiva: well ideally you guys would run the patch on your laptop
16:05:22 <asn> with that nice debug logging we were talking about
16:05:31 <asn> and you would use your laptop as normal, and maybe also use tor
16:05:37 <tdruiva> k
16:05:39 <asn> and then every now and then you would chekc your logs to see your guard usage
16:05:50 <asn> and ideally you should be happy about the security that your algorithm offers you
16:06:03 <asn> if you see the algo swithcing guards all the time, it's a problem.
16:06:12 <asn> if you see the algo moving to the 5th guard when your 1st guard is up, it's a problem.
16:06:13 <asn> etc.
16:06:18 <tdruiva> nice!
16:06:20 <asn> or well, that's what I'm planning to do at least :)
16:06:27 <tdruiva> :)
16:06:33 <rjunior> asn: I guess I have them on my computer
16:06:36 <rjunior> will send them nos
16:06:38 <rjunior> now
16:06:42 <asn> alright
16:06:47 <asn> iapazmino: indeed.
16:07:22 <asn> iapazmino: if you can find scenario-specific metrics that can be enlightining and we are missing
16:07:25 <asn> iapazmino: we should do them!
16:08:20 <iapazmino> asn: good, i'll think on this and run it by in the list for feedback
16:08:27 <asn> iapazmino: great.
16:09:03 <asn> for example, if we spend time on dystopic mode while we are on Default network. it's a problem.
16:09:07 <asn> we should never be down there.
16:09:08 <rjunior> asn: done
16:09:15 <asn> ok
16:09:20 <asn> any other questions?
16:09:32 <asn> otherwise, I will be monitoring the mailing list in the following days.
16:09:47 <asn> also I will take a look at the code just to see what's going on.
16:09:57 <iapazmino> asn: me done
16:10:06 <olabini> great
16:10:09 <olabini> thanks!
16:10:13 <asn> alright thanks guys!!!
16:10:14 <rjunior> ty asn
16:10:23 <asn> let's be in contact the following days :)
16:10:23 <iapazmino> thanks!
16:10:24 <asn> take care!
16:10:31 <iapazmino> good bye
16:10:38 <asn> #endmeeting