15:58:35 <shelikhoo> #startmeeting tor anti-censorship meeting
15:58:35 <MeetBot> Meeting started Thu May 12 15:58:35 2022 UTC.  The chair is shelikhoo. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:58:35 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
15:58:49 <shelikhoo> here is our meeting pad: https://pad.riseup.net/p/tor-anti-censorship-keep
15:58:49 <shelikhoo> feel free to add what you've been working on and put items on the agenda
15:58:51 <shelikhoo> Hi~
15:59:19 <meskio> hello
15:59:25 <itchyonion> hello
15:59:40 <arma2> dcf1: hey. i'm only a little bit here today (in in-person meeting), but: are there currently four flakeys running?
16:00:04 <dcf1> arma2: that's correct, four
16:00:40 <arma2> yay, thanks. i'm seeing about 21000 consensus fetches from russia per day per flakey,
16:00:48 <arma2> and about 12000 ip addresses from russia per day per flakey
16:01:07 <arma2> so we have maybe 50k to 80k snowflake users in russia in a day
16:01:32 <arma2> with many asterisks after the word maybe
16:01:53 <dcf1> I'm looking at bridge-stats-end 2022-05-12 05:10:44 (86400 s) and I see
16:02:07 <dcf1> bridge-ips ru=16544,us=2664,cn=1088,de=696,...
16:02:14 <dcf1> and in dirreq-stats-end 2022-05-12 05:10:35 (86400 s)
16:02:23 <dcf1> dirreq-v3-reqs ru=21528,us=2528,cn=1360,de=664,...
16:02:28 <dcf1> dirreq-v3-resp ok=31760,not-enough-sigs=0,unavailable=0,not-found=0,not-modified=3416,busy=0
16:02:29 <arma2> those stats are for *one* flakey, right?
16:02:38 <dcf1> that's for one instance, correct
16:02:41 <arma2> and each flakey keeps its own stats? ok great
16:03:06 * arma2 passes microphone back to meskio and shelikhoo
16:04:24 * shelikhoo wait one more minute for adding things to agenda
16:05:36 <shelikhoo> okay, now we move into discussion phase
16:05:41 <shelikhoo> the first topic is
16:05:42 <shelikhoo> Snowflake doesn't work in Russia (connection failure by timeout)
16:05:49 <shelikhoo> any detail about this?
16:06:17 <meskio> may ggus ?
16:06:22 <meskio> s/may/maybe/
16:06:28 <shelikhoo> I checked log from ru vantage point
16:06:52 <dcf1> per the bridge-stats quoted above, the great majority of snowflake users are from russia, so if it is true, it may only be inaccessible regionally
16:07:01 <shelikhoo> but no anomaly is discovered
16:07:18 <ggus> meskio: it wasn't me
16:07:45 <ggus> https://explorer.ooni.org/chart/circumvention?since=2022-04-12&until=2022-05-13&probe_cc=CN%2CIR%2CRU
16:07:49 <slacktopus> <hellais> FWIW OONI data shows many successful snowflake bootstraps from Russia: https://explorer.ooni.org/chart/mat?probe_cc=RU&test_name=torsf&since=2022-03-12&until=2022-05-13&axis_x=measurement_start_day&axis_y=probe_asn
16:08:01 <arma2> maybe the user who reported it was one of the "yellow" people in the ooni snowflake results?
16:08:03 <meskio> maybe a false positive
16:08:18 <arma2> that is, some fraction of snowflake attempts, from all over the world, seem to take too long
16:09:04 <shelikhoo> we need have a more detailed report
16:09:15 <shelikhoo> at least a copy and paste from tor log
16:09:22 <dcf1> I was looking at some of these results last week (since OONI stunreachability results recently became searchable) and found some puzzling features. I am planning to write an email later.
16:09:28 <shelikhoo> to understand why it stopped working
16:09:46 <meskio> yes, if the person that wrote it is not around I guess we can't do much
16:10:04 <dcf1> https://explorer.ooni.org/chart/mat?test_name=torsf&since=2022-04-13&until=2022-05-13&axis_x=measurement_start_day&axis_y=probe_cc
16:10:43 <dcf1> There's a higher proportion of anomalies from Canada than from Russia, which is in turn higher than China. It doesn't match intuition, at least.
16:11:18 <dcf1> Oh, I just realized the URL doesn't contain the country filter. I set the filter for CA+CN+IR+RU.
16:11:49 <meskio> italy also has a lot of anomalies
16:12:21 <meskio> maybe people on restricted nats on moments where there is not enough unrestricted proxies...
16:12:32 <slacktopus> <hellais> The many failures in the Italy results are caused by a bug in the test + a bug in the analysis. Most of those should be flagged as failed and ignored.
16:12:53 <dcf1> I'll send an email with a fuller summary, but from stunreachability measurements, it looks like 2 of the available STUN servers are blocked by DNS in Russia (one resolves to 127.0.0.1, one resolves to a known block IP). I found one on the Register but not the other.
16:13:18 <dcf1> Here's the CA+CN+IR+RU graph https://share.riseup.net/#TAfzUcgJ3Jhjr_carg5fsg
16:13:46 <arma2> does snowflake handle it properly if the stun server it picks is down, or if the stun server it picks turns out to not be a stun server? :)
16:14:02 <dcf1> Additionally, it looks like 1 STUN server is geoblocking users from Russia. Offhand, it doesn't seem like missing only 3 out of the pool of STUN servers would have a large effect on failure rates.
16:14:42 <dcf1> yes, unless there is a bug that prevents it from working as intended, it chooses a random subset of 50% of the pool, and tries each in turn until one works.
16:15:10 <arma2> ok
16:15:30 <meskio> how big is the pool of STUN servers?
16:15:39 <ggus> why sometimes snowflake works in this AS? https://explorer.ooni.org/search?since=2022-05-01&until=2022-05-13&probe_cc=RU&test_name=torsf&failure=false&probe_asn=AS25159
16:15:42 <dcf1> https://explorer.ooni.org/chart/mat?probe_cc=RU&test_name=stunreachability&since=2022-02-01&until=2022-05-13&axis_x=measurement_start_day&axis_y=domain
16:16:23 <dcf1> 12 STUN servers in the pool currently
16:16:31 <meskio> I see
16:17:19 <dcf1> stun.altar.com.pl:3478 is the one that looks like geoblocking, stun.stunprotocol.org:3478 and stun.voip.blackberry.com:3478 are the ones that look like Russian ISP blocking.
16:17:33 <dcf1> My notes are on another computer, but I will write up a summary and send it to the mailing list.
16:17:37 <shelikhoo> in theory, WebRTC can work as long as there is at least one working STUN server
16:18:00 <meskio> ggus: picking up one of those reports it failed at 10% 'Connected to a relay', weird
16:18:02 <dcf1> In summary though, my assessment is that STUN server blocking is not a major cause of Snowflake failures in Russia.
16:18:54 <dcf1> meskio: it always gets to at least 10%; 10% just means that snowflake-client sent a GRANT response to the SOCKS request, which it always does immediately. It does not mean it actually connected to a server or even successfully connected to the broker.
16:19:33 <dcf1> See discussion in an email thread starting 2022-03-28 "Subject: Re: Details about the OONI snowflake test?"
16:19:45 <meskio> ahh, true, I remember that
16:20:43 <meskio> would be nice to have the snowflake client log in the OONI tests
16:21:50 <dcf1> hellais: this MAT is really great, I like it a lot
16:22:02 <meskio> +1 <3
16:22:11 <ggus> +1
16:22:12 <shelikhoo> so we don't have any detailed reason for snowflake's failure... however, we do know that "connection failure by timeout" usually means the connection time out with a TCP server
16:23:08 <shelikhoo> or can it means ICE failure?
16:23:31 <shelikhoo> or failure at SCTP level?
16:24:24 <slacktopus> <hellais> <3
16:24:48 <shelikhoo> anyway, let's keep this topic and discuss it again next week after we got dcf1's email and the person that added this topic
16:25:20 <shelikhoo> anything more on this topic?
16:25:46 <shelikhoo> the next topic is
16:25:49 <shelikhoo> Moat gives blocklisted(ru) bridges in Russia (e.g. https://metrics.torproject.org/rs.html#details/1807BF9A521468998385F179DDBF928D2482A62C)
16:26:12 <shelikhoo> as far as I know, the blocklist isn't updated very frequently
16:26:28 <shelikhoo> so is this something expected?
16:27:54 <meskio> this is weird
16:28:02 <meskio> I'll open an issue to investigate it
16:28:10 <meskio> yes, this blocklist is basically never updated
16:28:57 <meskio> we did block every bridge in moat existing before december 2021 in russia
16:29:05 <meskio> and that is the only time we have used this mechanism
16:29:19 <meskio> but I'll investigate, there might be a bug in rdsys or bridgedb
16:30:17 <shelikhoo> this topic is submitted by the same author as last topic..,
16:30:27 <meskio> it looks like that
16:30:46 <shelikhoo> I hope there is more information on this, but maybe that's all we got
16:30:51 <shelikhoo> anything more on this topic?
16:31:01 <meskio> not from my side
16:31:43 <shelikhoo> okay, this is the last topic on the agenda, anything more we would like to discuss in this meeting
16:31:49 <meskio> the issue I created to track it: https://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/80
16:31:57 <shelikhoo> other than there is a reading group next week
16:32:12 <shelikhoo> We will discuss "Understanding the Impact of Encrypted DNS on Internet Censorship, ACM WWW 2021" on May 19th, 2022
16:32:12 <shelikhoo> https://shhaos.github.io/papers/www21-DoE.pdf
16:32:43 <ggus> meskio: i wonder how can we test that
16:33:19 <meskio> ggus: I can try from the vantage point, or set up some local testing
16:33:50 <ggus> ok!
16:34:24 <meskio> it could also be the user having an IP that is not listed as russian in our geoipdb
16:34:51 <shelikhoo> yes, IP->country is not always accurate
16:35:18 <meskio> and our list might be pretty old, is the one in polyanthum (old-stable debian package)
16:35:35 <meskio> soon will be updated to debian stable
16:36:19 <shelikhoo> one can reroute an IP to another location with BGP broadcast in a few minutes
16:36:51 <shelikhoo> so the old database will not be very useful
16:37:43 <meskio> yep
16:37:44 <shelikhoo> if necessary I don't see any obstacle for having a separate geoip database that don't depend on distribution
16:38:14 <shelikhoo> although that would means more manual updating
16:38:32 <meskio> yes, we can do that, AFAIK we do have debian packages of the geoipdb in deb.torproject.org
16:38:40 <meskio> we could use those, maybe we are already doing that
16:39:17 <shelikhoo> Yes. I think C tor is no longer depend on distribution for update
16:39:38 <shelikhoo> as we no longer have LTS
16:39:41 <meskio> I just checked, we are not, I'll ask TPA to use deb.tpo packages for it
16:40:27 <shelikhoo> I don't know if that would means some update for our anti-censorship systems...
16:41:11 <shelikhoo> but right now all of our docker and systems are mostly depend on distribution
16:41:23 <shelikhoo> not deb.tpo
16:42:06 <meskio> yes, we will need to revisit that at least for obfs4-bridge docker image
16:43:18 <meskio> actually obfs4-bridge does use deb.tpo
16:44:08 <shelikhoo> Yes...
16:44:30 <shelikhoo> Okay, anything more we need to discuss in this meeting?
16:45:08 <meskio> not from my side
16:45:27 <ggus> all good
16:46:46 <shelikhoo> #endmeeting