17:11:39 <ln5> #startmeeting snapshot #11
17:11:39 <MeetBot> Meeting started Mon Dec 16 17:11:39 2024 UTC.  The chair is ln5. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:11:39 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
17:11:51 <ln5> #topic agenda
17:12:15 <ln5> i have static updates and closing; more agenda items?
17:12:29 <ln5> s/static/status/1 :)
17:13:22 <ln5> wild agenda bashing today! keep calm people
17:14:35 <ln5> #topic status updates
17:14:57 <ln5> a second snapshot machine is in the makings; hw ETA: this week, best guess ETA for DSA: early january
17:15:53 <ln5> (the site will be in the same city as where snapshot-mlm-01 is, but a separate site. so naming...)
17:16:15 <ln5> iiuc lw07 is handling 10% of the incoming http traffic; mlm-01 is taking the rest (of what passes fastly)
17:16:33 <pkern[m]> woohoo
17:17:25 <ln5> i think that pkern[m] would like some more feedback on https://salsa.debian.org/snapshot-team/snapshot/-/merge_requests/28 before merging
17:17:32 <pkern[m]> As I said I'm unfortunately still distracted. I think there are two open PRs that need merging. One with DB changes, and one for the file attachment stuff.
17:17:48 <pkern[m]> I merged the apache2 config stuff that was required for the other PR, so that should be unblocked.
17:17:56 <ln5> yes, thanks
17:18:30 <ln5> would your work benefit from access to a test system?e
17:18:34 <pkern[m]> Well I guess we should go and merge that one and then have a followup to actually use the view. That would make lw07 faster and also make snapshot-mlm even faster. :)
17:19:01 <pkern[m]> Well, I already tested DB stuff on the live DB. /me hides. But locally I only tested with the unit tests. So I didn't test the code. So... yes, probably.
17:19:02 <ln5> sounds like it won't break anything. before we start using it.
17:19:22 <pkern[m]> It might break imports because queries are wrong, but we'd notice that, I guess.
17:19:27 <pkern[m]> *could be wrong
17:19:36 <ln5> i can test it out on -dev-01, but perhaps not this week
17:19:48 <pkern[m]> We are still not in a state where we can sanely reboot mlm-01 without an outage, even if lw07 will now pick up the traffic. I think.
17:19:58 <ln5> ok, good to know
17:20:54 <ln5> and from h01ger i heard earlier that the reprobuilder people seem happy enough
17:21:51 <ln5> i'm afraid i don't have an update on my action point of evaluating caching layers in containers yet
17:22:15 <pkern[m]> We are still serving failures when there are spikes, fwiw.
17:22:33 <pkern[m]> But with the scrapers neutered, it's only occasionally when something sends a lot of requests at once.
17:23:06 <ln5> are these failures unrelated to the db incosistency problem?
17:23:39 <pkern[m]> It's just temporary overload I think. I.e. we get spikes and then serve 503s.
17:23:51 <ln5> ic
17:23:52 <ln5> and how do we notice failures?
17:24:26 <h01ger> mind you, so far we only have 2 rebuilders. i hope to have 50 or more eventually. (debian should have 10, for for each arch, and then reproducible-builds.org should also have 10? and some more independent rebuilders...) ; tl;dr: i hope rebuilderd related traffic to snapshot.d.o will soon amount 5-10 times as much as we have now
17:24:39 <h01ger> s#for for#one for#
17:24:56 <ln5> h01ger: will they use a common (to them) cache?
17:25:29 <h01ger> i hope they will be spread over the world, different jurisdictions and different operators as well. so rather unlikely
17:25:45 <ln5> fair
17:25:46 <h01ger> they shall use local caches
17:26:33 <pkern[m]> I see stuff like https://ibb.co/FhzWNP4
17:26:56 <pkern[m]> Fastly is caching on the edge nodes and not centrally right now.
17:27:18 <pkern[m]> Other 5xx is the scraper blocker
17:27:37 <h01ger> but hey, if there's one for each arch in half a year, i'm happy too. (and only the amd64 rebuilder builds all 70k binary packages from a current suite. the other arch rebuilders only build their 30k arch:any packages.)
17:28:02 <pkern[m]> I expect rebuilding to be the same active set rather than requesting uncommon content.
17:28:17 <h01ger> pkern[m]: yes
17:28:45 <ln5> h01ger: do you generally know a checksum for the packages you need from snapshot.d.o.?
17:28:50 <h01ger> rebuilding a suite (with 70k binary packages) needs about 30k different binary packages in 100k variations in total.
17:29:17 <pkern[m]> https://ibb.co/z88vNvF looks like the scrapers gave up
17:29:25 <h01ger> ln5: we had this discussion (apt requesting by sha256 sum) before, but i'm atm not sure where exactly this is stuck
17:30:53 <ln5> h01ger: would you like to try to figure out a good place to have that discussion?
17:30:54 <h01ger> pkern[m]: this was re: "I expect rebuilding to be the same active set rather than requesting uncommon content" - rebuilding a suite (with 70k binary packages) needs about 30k different binary packages in 100k variations in total. so, yes, thats a very small subset of snapshot we are interested in. 100k debs making up roughly 100gb size. (for one arch)
17:31:52 <pkern[m]> So doesn't quite fit into RAM but also not that far fetched.
17:32:12 <pkern[m]> (snapshot-mlm having 500G)
17:32:19 <h01ger> ln5: i fear i'm involved in too many discussions already when i discussed this with juliank (apt maintainer) this summer there were some blockers which made it so unlikely that i didnt even file a wishlist bug.
17:32:21 <pkern[m]> So I'm not really worried about that.
17:32:34 <h01ger> ln5: i fear i'm involved in too many discussions already. when i discussed this with juliank (apt maintainer) this summer there were some blockers which made it so unlikely that i didnt even file a wishlist bug.
17:32:56 <pkern[m]> And the redirects are very highly cachable
17:33:13 <ln5> h01ger: ok, np. thanks for bringing it up with apt maintainer.
17:33:48 <ln5> pkern[m]: snapshot could cache them forever, yes?
17:33:59 <h01ger> ln5: i'll also keep having this on my mind, because it seems the right thing todo, even though current design doesnt seem to allow it
17:34:33 <ln5> h01ger: current design of snapshot you mean? bc it lacks sha256?
17:35:20 <h01ger> no. i think i mean debian archive.
17:35:41 <ln5> ic. yes, that's a bit worse.
17:37:40 <ln5> any other status updates? or other agenda items?
17:38:18 <ln5> if not, we'll move to closing
17:39:01 <ln5> #topic closing
17:39:16 <ln5> for the next meeting i propose 2025-01-20T18:00:00Z
17:39:25 <ln5> that's one hour later than today's meeting
17:40:37 <ln5> more suggestions?
17:40:54 <ln5> #agree next meeting 2025-01-20T18:00:00Z
17:41:02 <ln5> thanks all
17:41:04 <ln5> #endmeeting