16:02:25 <meskio> #startmeeting tor anti-censorship meeting
16:02:25 <MeetBot> Meeting started Thu Jun  8 16:02:25 2023 UTC.  The chair is meskio. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:02:25 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
16:02:38 <meskio> sorry, I was starting the meeting in our #tor-anticensorship channel
16:02:54 <meskio> and wondering why the meetbot was not talking XD
16:03:11 <shelikhoo> hi~
16:03:13 <shelikhoo> hahaha
16:03:15 <meskio> here is our meeting pad: https://pad.riseup.net/p/tor-anti-censorship-keep
16:03:24 <meskio> feel free to add what you've been working on and put items on the agenda
16:03:35 <meskio> hello everyone!!!!
16:04:39 * meskio needs to figure a key convination for :face_palm: emoji in the terminal, will use it frequently
16:04:51 <itchyonion> hello
16:05:19 <meskio> anything to discuss about "Report of TLS-in-DTLS detection and throttling in China that affects Snowflake"? (is from last week)
16:05:19 <shelikhoo> X~X <- shell's version
16:05:29 <shelikhoo> yes
16:05:30 <meskio> that is a good one X~X
16:05:44 <shelikhoo> I have tested and the result is inconclusive
16:06:03 <dcf1> I wrote a summary in the pad, shelikhoo I hope I have the details right
16:06:14 <shelikhoo> because we change the provider of the vantage point(underlying one)
16:06:37 <shelikhoo> the network environment changed, and 20%+ packet loss is not reproduced
16:07:32 <dcf1> The test was meant to see if the traffic analysis signature of the beginning of the DTLS connection was leading to throttling (as https://github.com/net4people/bbs/issues/255 reported), but in this test, both with and without padding were equally fast
16:07:35 <meskio> ohh, so now there is no packet loss and snowflake works fine?
16:07:54 <dcf1> so we don't know if the DTLS packet loss is a regional thing, or if it stopped, or what
16:07:57 <shelikhoo> we didn't reproduce the issue described in the issue
16:08:26 <meskio> but you still see the old packet loss we were seeing?
16:08:45 <shelikhoo> no, and that could because the vantage point instance was recreated
16:09:28 <meskio> I see, so we don't know if we saw two different problems of packet loss or the same one
16:09:43 <shelikhoo> yes...
16:10:02 <shelikhoo> and the question now is whether we wants to go ahead and introduce padding
16:10:10 <shelikhoo> or wait for more information
16:10:41 <meskio> what are the drawbacks of introducing the padding?
16:10:44 <dcf1> I can update the BBS thread to say that we were unable to reproduce the DTLS throttling and see what the original reporter has to say
16:11:09 <shelikhoo> the network environment in China vary by region and ISP
16:11:40 <shelikhoo> so it not unexpected that some network environment have additional restrictions
16:11:56 <dcf1> The padding is only to disguise the first few (<10) packets in either direction, to obliterate the encapsulated TLS signature. So it doesn't add any overhead past the beginning of the connection.
16:12:15 <dcf1> We will have to think of a smart way to do it, what I did in my patch was just a quick job.
16:13:02 <dcf1> It's easy enough to change the signature so it no longer resembles tunnelled TLS, but we have to be careful not to introduce a new traffic fingerprint at the same time.
16:13:50 <dcf1> The BBS reporter also suspected it may have something to do with KCP restransmission dynamics.
16:14:51 <shelikhoo> I suspect it is a ISP specific restriction
16:15:38 <shelikhoo> since I run a KCP over webrtc proxy myself from my home in China, and no interference was observed
16:15:55 <dcf1> The only corroborating evidence I know is a reduction in bandwidth on snowflake-02 starting 2023-05-22: https://metrics.torproject.org/rs.html#details/91DA221A149007D0FD9E5515F5786C3DD07E4BB0
16:16:14 <dcf1> (With no corresponding decrease in users.)
16:16:42 <dcf1> Bandwidth statistics are not broken down by country unfortunately, so we cannot say if it was due to China specifically.
16:18:11 <dcf1> Personally I don't think it's an immediate task, but shaping at least the first couple of KB of a connection is something we should be thinking about
16:18:55 <dcf1> At this point it likely doesn't have to be full-blown website fingerprinting defenses, all the discussion I've seen has been just about the first few packets (size, timing, direction, burst).
16:19:11 <shelikhoo> yes, I think this shouldn't be a huge issue if we are adding a few kbs of padding
16:19:23 <shelikhoo> is the users are already using snowflake
16:19:35 <meskio> +1
16:19:37 <shelikhoo> we just need to make sure there isn't new fingerprint added
16:20:32 <dcf1> Also if directionality plays a role, we could possibly mitigate the problem at the bridge as well,
16:21:09 <dcf1> by having the bridge send a chunk of padding even before it receives a tls client hello. My guess is that "server speaks first" would be enough to break a classifier.
16:21:43 <shelikhoo> but it would require some design change...
16:22:06 <shelikhoo> as right now it only start handle traffic after receiving clientid
16:22:18 <dcf1> The encapsulation package in snowflake has encapsulation.WriteData and encapsulation.WritePadding. Currently WritePadding is totally unused, and WriteData is only called on demand. But we could insert WritePadding wherever we want, and change the timing or segmentation of WriteData.
16:23:12 <dcf1> I don't think it would be a difficult change. Just suggesting the idea because that may be an quicker mitigation than deploying new clients.
16:23:40 <dcf1> The server doesn't need to know a client ID before calling encapsulation.WritePadding, I don't think.
16:24:07 <shelikhoo> that's true...
16:24:10 <dcf1> SO it can send as much phony padding as it wants at the beginning of the connection.
16:25:31 <dcf1> basically insert a WritePadding somewhere in turbotunnelMode: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/blob/130b63ccddb64b10a881b150c93c0122e0698a9e/server/lib/http.go#L143
16:25:46 <shelikhoo> yes, however without the ability to reproduce the issue, it is hard to evaluate its effectiveness
16:25:59 <dcf1> Yes
16:26:13 <dcf1> I just want to throw some ideas out there so we're ready with a plan in case it clearly becomes necessary
16:27:02 <meskio> what will be the path forward? create an issue to collect all this information and brainstorm on how to do the padding in the future? integrate the existing patch and see what is the next move on the cat and mouse game? do a bridge fix and see if drop of users recovers?
16:27:40 <dcf1> (There was no drop of users, only of bandwidth.)
16:27:57 <meskio> true, bandwidth
16:28:40 <shelikhoo> we could wait for comment from author's issue before our next move
16:29:02 <shelikhoo> let's say we could ask what network environment can be used to reproduce that
16:29:15 <dcf1> I'll post an update to https://github.com/net4people/bbs/issues/255
16:29:47 <meskio> sounds good to me
16:31:55 <meskio> anything else on this topic?
16:31:55 <meskio> let's move to the next one
16:31:55 <meskio> "meek-azure deprecation'
16:31:55 <meskio> azure is going to shutdown their domain front in 5 months, we need a plan on what to do with it
16:31:55 <meskio> AFAIK meek users are mostly from Turkmenistan and china
16:31:55 <meskio> for china we are working on improving snowflake, and meek is on heavily used compared with snowflake or obfs4, so I think is fine
16:31:55 <meskio> but for Turkmenistan we don't have a great path forward
16:32:29 <meskio> most people uses obfs4 bridges, but they get blocked fast and only few of them are actually reachable, our community team is doing an amazing work on finding working bridges and handing them manually to people on the country
16:32:44 <meskio> ~900 users rely on meek to connect to tor
16:33:22 <shelikhoo> we could try to improve snowflake if we could got a vantage point, or somehow have access to their network environment
16:33:52 <gaba> is any of the new PTs or distributiom methods coming out like conjure, webtunnel or lox something that will work there?
16:34:03 <meskio> cohosh: I wonder if conjure is mature enough to test it there and see if it works, Turkmenistan is blocking IP ranges easily so not sure how well it will work, but maybe an option to try out conjure with more users :)
16:34:25 <cohosh> yeah we can talk to gus about that
16:34:28 <meskio> shelikhoo: the problem with snowflake is that many home IP ranges are blocked from TM, so snowflake fails to connect
16:34:48 <shelikhoo> yes... and I think they have blocked stun as well
16:35:41 <meskio> we could set up meek-fastly, but will be great to find something were we don't need to pay a big bill and to don't burn fastly and get them to block domain fronting too
16:36:07 <meskio> anyway, we don't need to find a solution yet, but I'll be trying to have a plan with time in advance
16:36:56 <shelikhoo> gaba: they block PT by ip address block, so I don't think any new pt is specialized with deal with that...
16:37:39 <meskio> cohosh: great, I think we can find people in TM to try conjure and see if it works, and if so we can think what are the next steps (we could add it to TB stable without UX and enable it with circumvention settings only for TM)
16:38:17 <shelikhoo> I imagine there is nothing preventing them from blocking ISP hosting the conjure
16:38:25 <cohosh> yep, i'm skeptical it will work but it's worth a try
16:38:37 <nina13[m]> May I jump in? I share an info about conjure in tm, so hope to provide some feedback soon
16:38:41 <meskio> yes, I agree it might not work
16:39:10 <meskio> nina13[m]: thank you for comming here, I was thinking on poking you about that :)
16:39:23 <meskio> so you did already ask some people to try it out?
16:39:43 <meskio> let's see how it goes
16:40:55 <shelikhoo> yep
16:41:47 <cohosh> nina13[m]: thanks!
16:41:55 <meskio> anything more on this topic?
16:42:17 <nina13[m]> meskio: Yes, try to reach out users in tm and russia
16:42:48 <meskio> thanks, let me know if I can help there
16:43:09 <onyinyang[m]> I am missing some context on how the obfs4 bridges are being blocked and the IP address blocking, are there issues or previous discussions on this I could look into?
16:43:23 <onyinyang[m]> Also happy to discuss more in #tor-anticensorship:matrix.org after the meeting
16:43:50 <meskio> onyinyang[m]: TM is being blocking big IP ranges, like for example all hetzner (a common VPS provider used by bridges) is blocked
16:44:02 <meskio> they don't care if a big chunk of internet is not reachable
16:44:09 <shelikhoo> and they are blocking a lots of, like cloudflare is mostly blocked
16:44:13 <meskio> is different than china that does try to keep most internet accesible
16:44:33 <shelikhoo> in china is mostly accessible but slow...
16:44:48 <meskio> yes, different...
16:44:49 <onyinyang[m]> yeah, I was looking into that on my own a bit. I'm sort of surprised that meek/azure is accessible with that context in mind.
16:45:13 <meskio> I guess TM's government uses teams???
16:45:26 <onyinyang[m]> lol I was just thinking that XD
16:45:58 <meskio> let's move on
16:46:11 <meskio> I guess we have discussed already the packet loss in china
16:46:16 <shelikhoo> yes..
16:46:41 <meskio> great, then let's go to the last topic the webtunnel documentation for bridge operators
16:46:52 <meskio> https://pad.riseup.net/p/6hiwvSWJanxml7DS299z
16:46:55 <meskio> shelikhoo???
16:47:14 <shelikhoo> yes! I have write the document and it ready for discussion'
16:47:24 <shelikhoo> anything missing from it or any advise?
16:48:04 <meskio> something about how to get your bridgeline and test it?
16:48:35 <shelikhoo> oh, yes...
16:48:56 <shelikhoo> I could try to build this into the container
16:49:30 <meskio> I think many people uses docker-compose, I wonder if is not better to produce a docker-compose file
16:49:45 <meskio> is more readable than the docker run flags :)
16:50:07 <shelikhoo> the issue with docker compose is that the file will need to be generated
16:50:19 <shelikhoo> as the OR port and bridge name is generated
16:50:21 <meskio> you only need to generate a .env
16:51:01 <shelikhoo> oh! I will try to write a docker compose, and describe how to construct the bridgeline
16:51:18 <shelikhoo> and see if I could automate the generation of the bridge line
16:51:30 <meskio> sounds great
16:51:40 <meskio> thank you for the work, it looks pretty good
16:52:13 <shelikhoo> yes! no problem!
16:52:56 <shelikhoo> EOF from me
16:53:07 <meskio> ok, then we can move on
16:53:27 <meskio> I see there is an update on opencollective's snowflake-01: https://opencollective.com/censorship-circumvention/projects/snowflake-daily-operations/updates/2023-may-update
16:53:59 <meskio> anything more to discuss?
16:54:07 <shelikhoo> EOF from me
16:54:58 <meskio> cool, we can finish the meeting, 3 mins before the our :)
16:55:02 <meskio> #endmeeting