19:02:10 <MadameZou> #startmeeting
19:02:10 <MeetBot> Meeting started Wed Jul 20 19:02:10 2011 UTC.  The chair is MadameZou. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:02:10 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
19:02:18 <zobel> Hi
19:02:22 <MadameZou> #topic "Ask the DSA Team"
19:02:59 <zobel> if you wonder who we are, DSA are the ones behind guys, that are running all the debian.org machines
19:04:07 <zobel> that is currently Peter Palfrader (weasel), Stephen Gran (sgran), Luca Filipozzi (luca) and me (martin Zobel-Helas)
19:04:44 <zobel> we are responsable for all the debian.org machines, are applying security-upgrades, upgrading all the machines, setting up new ones.
19:05:11 <zobel> basicly we are the ones with "root" on all of those machines.
19:05:39 <zobel> so, weasel, do you want to add something?
19:05:47 <zobel> or should we start with the Q&A?
19:06:20 <weasel> not particularly; works for me
19:06:37 <weasel> zobel: do we have the slides from 2009 up on the web somewhere?
19:06:53 <MadameZou> < ruipb> Question: What does it takes in terms of programming and knowledge to became a DSA developer?
19:07:29 <zobel> well, you should be able to understand what code does, but we are more doing sysadmin'ing
19:08:02 <weasel> Being a sysadmin isn't really a programming job.  You have to be able to script stuff, but that's not really the same as a coding project.
19:08:15 <zobel> it is helpful if you can write some of the scripting languages, like, perl, python. but no need for that
19:08:20 <zobel> next
19:08:27 <weasel> being able to fix broken stuff comes in useful from time to time tho.
19:09:00 <MadameZou> < jespada> QUESTION: What kind of tool(cfengine, puppet, chef) DSA's use to automate deploys/install apps/boxs ?
19:09:30 <zobel> we are using puppet
19:09:44 <weasel> we have puppet, our checklists (mostly linked off dsa.d.o), and userdir-ldap
19:09:48 <zobel> the config for that is public. on http://anonscm.debian.org/gitweb/?p=mirror/dsa-puppet.git;a=summary
19:10:02 <Rhonda> #link http://anonscm.debian.org/gitweb/?p=mirror/dsa-puppet.git;a=summary
19:10:41 <zobel> you might want to have a look on the slides we did for DC9 DSA BoF: http://people.debian.org/~zobel/dc9.pdf
19:10:44 <zobel> next?
19:10:55 * h01ger just wants to thank the DSA team for awesome work throughout the years
19:11:04 <MadameZou> < pabs> QUESTION:how many hours per day do they work? does the DSA team have a list of things for potential new team members to fix or (re-)implement?
19:11:09 <zobel> h01ger: thanks, and you are welcome.
19:11:17 <MadameZou> ehm ops
19:11:25 <MadameZou> I've pasted two in one
19:11:28 <pabs> :)
19:11:35 * jespada tks for  answer
19:11:38 <Rhonda> #link http://people.debian.org/~zobel/dc9.pdf
19:11:40 <MadameZou> < pabs> QUESTION: does the DSA team have a list of things for potential new team members to fix or (re-)implement?
19:11:45 <MadameZou> this is the right one :D
19:12:24 <weasel> well, there's always usedir-ldap.  zobel knows more about the state of improving that one
19:12:32 <zobel> well, there are few things that need to be done potentialy. like some changes to ud-ldap
19:13:03 <zobel> also i would like someone to setup or guide us through setting up a OAUTH2 provider for debian.org
19:13:21 <zobel> so we can have all DDs having accounts to rt.debian.org or other web-apps.
19:13:27 <zobel> that need password protection
19:13:30 <zobel> like wiki
19:14:06 <zobel> we usually find things that need to be done.
19:14:39 <zobel> so if you guys out there find something that annoys you, try to work as far as you get, and the ask DSA to do the last steps that need root access.
19:14:52 <pabs> followup: do you have a place to document the need for such things to be done?
19:15:09 <zobel> we are happy if we don't need to admin every single service inside debian
19:15:12 <zobel> next.
19:15:15 <weasel> oauth hasn't been in the queue for long yet,
19:15:32 <zobel> pabs: rt.debian.org
19:15:40 <weasel> and we have http://dsa.debian.org/poneys/, but I'm not sure how current it is
19:15:52 <zobel> they are probably not.
19:16:01 <weasel> well, the mail thing is still kinda on the list
19:16:06 <zobel> but we can add some more during DC11 i think
19:16:11 <weasel> the security stuff might be mostly done.  or at least to a sufficient degree
19:16:36 <zobel> next
19:16:42 <MadameZou> <enrico> how do you decides who fixes something? How do you avoid the "someone else will do it" problem of online teams?
19:17:08 <zobel> enrico: that happens from time to time
19:17:22 <zobel> but most times i tell on the admin channel what i am working on
19:17:28 <zobel> so the other members know
19:17:53 <MadameZou> <MadameZou> are configuration files for debian machines public? and if so where can we find/browse them?
19:17:54 <weasel> and we are a small enough team and in similar timezones.
19:18:05 <weasel> everything that is in puppet is public.
19:18:13 <zobel> that was already answered: http://anonscm.debian.org/gitweb/?p=mirror/dsa-puppet.git;a=summary
19:18:19 <MadameZou> ops, sorry
19:18:22 <MadameZou> <enrico> are Debian machines IPv6 ready? Are all services reachable and working from IPv6?
19:18:29 <weasel> things that aren't, are only accessible to people who have shells on the hosts in question, if the files are world readable
19:18:34 <zobel> its public, i think it is something like 10min behind.
19:18:49 <weasel> a lot of machines have ipv6, but not all.
19:18:55 <weasel> and no, probably not.
19:18:55 <zobel> enrico: that heavyly depends on the hosters.
19:19:08 <zobel> we have more than 50% ready
19:19:24 <zobel> but we do not have all machines on IPv6
19:19:33 <zobel> next.
19:19:36 <MadameZou> < morphic> do you guys use some virtualization technology? or physical machines in the farm?
19:19:47 <weasel> kvm+libvirt.
19:20:09 <morphic> cool
19:20:19 <zobel> morphic: as weasel said kvm + libvirt on powerful machines with lots of storage.
19:20:20 <weasel> [ http://anonscm.debian.org/gitweb/?p=mirror/dsa-kvm.git;a=summary ]
19:20:22 <zobel> next
19:20:25 <MadameZou> why have you stopped using Monkeysphere?
19:20:30 <weasel> we never used it.
19:20:32 <weasel> so "mu"
19:21:12 <zobel> next
19:21:15 <MadameZou> How many machines do you administrate?
19:21:22 <weasel> zobel: I wasn't really done yet :)
19:21:37 <zobel> oh
19:21:42 <pijanc> :)
19:21:45 <zobel> weasel: then please, go ahead
19:21:46 <weasel> the ssh host key fingerprints can be downloaded from db.d.o, and from /etc/ssh on all debian.org machines.  people can just place them in ~/.ssh/
19:21:54 <weasel> as known_hosts2 or append to their known_hosts.
19:22:02 <weasel> also, we have ssh host key fingerprints in DNS (SSHFP)
19:22:09 <weasel> and debian.org dns is secured.
19:22:20 <weasel> now I'm done :)
19:22:37 <MadameZou> How many machines do you administrate?
19:22:48 <MadameZou> (second try) ;)
19:22:56 <zobel> we have around 130 machines running atm
19:23:00 <MadameZou> wow
19:23:00 <weasel> weasel@draghi:~$ ldapsearch -LLL -ZZ -x  -h db.debian.org 'host=*' hostname | grep debian.org | wc -l
19:23:00 <weasel> 135
19:23:13 <MadameZou> < Bombenleger> Question: How do you guys install security patches on the maschines? Login in every single one and do an apt-get dist-upgrade?
19:23:13 <mehdi> physical/virtual?
19:23:13 <weasel> (everybody can run that query)
19:23:32 <zobel> in dozens of datacenters.
19:23:32 <weasel> weasel@draghi:~$ ldapsearch -LLL -ZZ -x  -h db.debian.org 'host=*'  | grep -i phy | wc -l
19:23:32 <weasel> 34
19:23:35 <weasel> 34 of them are VMs
19:23:46 <weasel> weasel@draghi:~$ ldapsearch -LLL -ZZ -x  -h db.debian.org 'host=*'  purpose | grep -i kvm | wc -l
19:23:47 <weasel> 6
19:23:50 <weasel> on 6 pieces of iron.
19:24:06 <weasel> ok, re security updates.
19:24:14 <weasel> yes, basically we run apt-get update && apt-get upgrade on all machines
19:24:30 <Bombenleger> manually?
19:24:36 <weasel> it's in a shell for loop.
19:24:47 <zobel> weasel is doing that mos of the times, so he is the better person to answer that.
19:24:55 <weasel> with a bit of a hack so that you can run it like 40 times in parallel, without races
19:25:06 <zigo> Like: for i in <list-of-machines> ; do apt-get update ? Something like that?
19:25:21 <zobel> zigo: upgrade
19:25:36 <weasel> http://asteria.noreply.org/~weasel/volatile/2011-07-20-Vv1TDcddWT0/file6lTu98
19:25:37 <zobel> update is done by cron, so nagios can notify us of outstanding updates.
19:25:38 <weasel> like that, yes.
19:26:11 <zobel> i am done
19:26:15 <weasel> for i in $DEBHOSTS; do mkdir hosts/$i || continue; ssh $i "sudo <apt-get stuff>"; done
19:26:26 <Bombenleger> ok i got it. thank you!
19:26:27 <weasel> the mkdir || continue allows me to run the script any number of times
19:26:34 <weasel> (mkdir is atomic)
19:26:41 <weasel> then, done :)
19:26:42 <weasel> next
19:26:44 <MadameZou> < babilen> QUESTION: What are the biggest challenges in administrating such a global service? How is data shared between hosts? (think AFS)
19:27:03 <weasel> we have had an AFS for half a year or so now.  with no real use case for it
19:27:07 <weasel> so we will get rid of it again
19:27:19 <weasel> (also, AFS doesn't really perform too well over the Internet)
19:27:44 <zobel> the biggest challenge is to keep the users (and DDs) happy
19:27:48 <weasel> one of the biggest challenges is probably dealing with a great number of different places
19:28:03 <weasel> and some have weird firewall restrictions,
19:28:10 <weasel> and in others we are even behind NAT
19:28:15 <weasel> and that makes things just painful.
19:28:16 <zobel> if noone complains on not running services, we have done a good job.
19:28:39 <weasel> having all the hardware in 4 or 5 places would be nice.  we're probably an order of (decimal) magnitude off of that number tho
19:29:14 <zobel> we have too many hosting locations IMHO
19:29:23 <weasel> we do.  it's getting better, but slowly.
19:29:37 <zobel> but that can't change that fast.
19:29:54 <zobel> we are heavyly depending on hosters here.
19:30:03 <zobel> done
19:30:08 <MadameZou> < daemonkeeper> Question: Where does hardware running various Debian services come from? Is there demand for hosting/housing facilities?
19:30:08 <weasel> as for data sharing, there's two things.  user account information gets pushed/pulled from db.debian.org
19:30:27 <weasel> and there's puppet.  DSA doesn't sync any of the other data.  mirroradm does some of that.
19:30:51 <weasel> some of it we get donated, others we have bought (in the last one or two years)
19:31:12 <zobel> HP was donating a good number of machines in last years.
19:31:27 <zobel> but that is not current any more.
19:31:45 <weasel> as for housing, a single machine or two probably isn't useful.  unless it's in some place like africa or asia for a security mirror.
19:32:00 <zobel> we still get donations
19:32:04 <weasel> we could maybe do with a place that's willing to host half a rack of stuff or more in the long run
19:32:18 <zobel> for housing: a rack or two are a good thing! :)
19:32:46 <weasel> done?
19:32:49 <daemonkeeper> Thanks for your answers, yes.
19:33:14 <zobel> we would like to avoid opening new housing/hosting locations unless we can move a bigger number of machines to there.
19:33:25 <zobel> and the moving also costs money
19:33:47 <zobel> which leader@d.o needs to ack before we do that.
19:33:48 <zobel> done
19:33:55 <zigo> Please define "bigger number of machines"
19:34:03 <zigo> 5? 10 ? 40 ?
19:34:35 <zobel> well, if we can fill a rack with it, yes 10, 20.
19:34:45 <zobel> and it should have decent bandwith
19:34:48 <zobel> done
19:34:54 <MadameZou> < tiago> MadameZou: QUESTION: are DSA people working on debian.net/debconf.org machines? Or any cooperation between them?
19:35:15 <tiago> aaaqweqwe3
19:35:17 <weasel> they aren't our turf.
19:35:30 <weasel> there is some overlap between debconf and debian admin (sgran is in both teams),
19:35:38 <zobel> tiago: basicly: no.
19:35:39 <tiago> oops sorry
19:35:44 <weasel> and at debconfs members of DSA have historically helped with the admin work on site,
19:36:03 <weasel> and we export ldap data to some of the debian.net porter machines, but we don't maintain them
19:36:14 <weasel> zobel: [I'm done]
19:36:17 <zobel> me too
19:36:20 <weasel> next
19:36:28 <MadameZou> < daemonkeeper> Question: In your opinion, dies it make life easier or complicated, having technically skilled DDs as "customers" on machines? :)
19:36:33 <daemonkeeper> *does
19:37:01 <weasel> people need to realize that having installed a debian or two for themselves at home doesn't make them sysadmins :)
19:37:25 <weasel> other than that, having users with Clue probably is a plus
19:37:51 <weasel> at least I think it is
19:38:19 <weasel> zobel: anything else?
19:38:25 <MadameZou> ehm, zobel is having some problems with connection
19:38:32 <MadameZou> ok, is back :D
19:38:43 <zobel> daemonkeeper: it helps if people can express what they awant like: "please install foo" or "do bar .."
19:38:57 <zobel> done
19:38:58 <MadameZou> < ansgar> QUESTION: How many machines (and what sort of) have root rights for non-DSA?
19:39:17 <weasel> ( sometimes it's difficult to explain to them however, that we won't apt-get install <random service providing daemon> for them :)
19:39:38 <weasel> the buildd people have root on the buildds to maintain their chroots,
19:39:49 <weasel> with the understanding that they don't touch any of the / system
19:40:07 <zobel> ansgar: on very few porter machines some porters have access to root.
19:40:24 <zobel> but that is IIRC currently only 2 machines.
19:40:31 <weasel> and local admins have root on machines at their place sometimes
19:40:38 <weasel> (usually the local admins are DDs too)
19:40:49 <zobel> i am done
19:40:50 <weasel> again, they are expected to never use it
19:40:58 <weasel> done
19:41:03 <zobel> only for stuff like shutdown
19:41:11 <zobel> in case of emergency
19:41:14 <weasel> yup
19:41:31 <MadameZou> < rambominator> QUESTION:  What strategy do you use for backups, which backup tools?
19:41:47 <zobel> some have ssh-root access to command /sbin/shutdown
19:41:50 <weasel> Joey(iirc) wrote da-backup a couple years ago
19:41:55 <weasel> da-backup is yet another wrapper around rsync
19:42:06 <weasel> we use that for backing up some directories on some hosts
19:42:19 <weasel> (see /etc/da-backup on a host to see if we back up anything from it)
19:42:32 <zobel> we mostly only backup services and /etc
19:42:47 <zobel> we can install most machines just using D-I
19:42:53 <weasel> or debootstrap
19:43:00 <zobel> so we can ignore the OS mostly for backup
19:43:30 <weasel> we probably should do some full backups for a few core machines (like db),
19:43:34 <zobel> and have virtual packages like debian.org-www-master.d.o to re-install a machine like www-master.d.o
19:43:54 <weasel> so that we can have disaster recovery stuff.  but we don't have that right now, and we don't really know what we would use
19:44:09 <weasel> zobel: s/virtual/meta/?
19:44:12 <zobel> which just pulls in the right depends.
19:44:15 <zobel> weasel: err, yes.
19:44:28 <zobel> learn: the weasel is always right! :)
19:44:41 <zobel> i am done
19:44:46 <weasel> same
19:44:47 <MadameZou> < alex_muntada> QUESTION: how do you share sysadmin secrets? gnupg, keepassx, etc.
19:44:57 <weasel> pws
19:45:05 <weasel> a gpg wrapper I wrote two years ago
19:45:22 <zobel> alex_muntada: there is a tool written by weasel called pwstore (pws) that is a wrapper around gpg
19:45:28 <weasel> git clone http://svn.noreply.org/git/pwstore.git
19:45:33 <weasel> http://asteria.noreply.org/~weasel/volatile/2011-07-17-k7Dj2x4W6WE/README.asciidoc
19:45:38 <zobel> it knows about groups
19:45:51 <zobel> so you can share stuff with persons other than DSA.
19:45:56 <zobel> like with local admins.
19:46:13 <zobel> so it basicly encripts for all persons in that group.
19:46:20 <zobel> encrypts
19:46:25 <zobel> i am done
19:46:28 <weasel> .
19:46:40 <mehdi> #link git clone http://svn.noreply.org/git/pwstore.git
19:46:51 <MadameZou> QUESTION: How many of the administered machines have KVM over IPs? In your opinion, does having access to a KVM over IP very  important for remote administering?
19:47:04 <n0rman> aaaaaaaaaaaaaaaaaaaa
19:47:09 <weasel> most machines have iLO or some form of remote management
19:47:18 <zobel> yes it helps.
19:47:22 <weasel> for some it's only serial console and remote power
19:47:31 <MadameZou> ops! this one is from zigo
19:47:37 <zobel> this way we do not need remote hands in most cases.
19:47:39 <zobel> zigo: ^
19:47:48 <MadameZou> (sorry zigo, I've done a copypaste mess ;) )
19:47:49 <weasel> a few machines have neither, and that's mostly for older machines.  we tend to make management access a prerequisite for new machines
19:47:54 <zobel> only for swapping broken hardware.
19:48:19 <zobel> done
19:48:23 <weasel> it's not that we need it often,
19:48:28 <weasel> but when we do, we really need it
19:48:44 <zobel> we only needed a lot of local admins at once...
19:48:45 <weasel> like when a kernel upgrade fails or weasel kills sshd on all machines
19:49:00 <weasel> .
19:49:17 <zobel> when a config change with puppet break most of our machines.....
19:49:28 <zobel> but that happens VERY seldomly..
19:49:32 <zobel> .
19:49:37 <MadameZou> < pabs> QUESTION: are there many pieces of software that are not packaged for Debian running on Debian servers? are they installed using local packages or manually installed? what type of software is most of that?
19:50:00 <weasel> there's two groups
19:50:04 <zobel> pabs: that is mostly for services.
19:50:17 <weasel> there's stuff that users/roles install.  that's not our turf.  (/srv/$team/<stuff>)
19:50:38 <weasel> for DSA, the only non-packaged stuff I can think of is various raid monitoring tools that just live in /usr/local/
19:50:51 <zobel> .
19:50:57 <weasel> everything else is either shipped via puppet or packaged and in the apt repository on db.d.o
19:51:04 <weasel> it's not necessarily in debian
19:51:29 <weasel> some software is pulled in from other non-standard repositories
19:51:41 <weasel> e.g. the buildd people have their own repos which is enabled on our buildd machines
19:51:47 <weasel> .
19:51:53 <MadameZou> < tiago> MadameZou: QUESTION: are DSA people working on debian.net/debconf.org machines? Or any cooperation between them?
19:51:54 <zobel> sometimes we pull stuff from backports
19:51:55 <zobel> if needed
19:52:02 <weasel> MadameZou: we already answered that
19:52:08 <MadameZou> ops sorry
19:52:29 <MadameZou> how many calls do DSAs get a week?
19:52:30 <zobel> next!
19:52:36 <weasel> calls?  as in phone calls?  none.
19:52:38 <zobel> what kind of calls?
19:53:02 <enrico> zobel: like, issues that come up for handling
19:53:07 <MadameZou> LOL @ phone calls
19:53:08 * zobel calls the weasel sometimes  :)
19:53:12 <enrico> zobel: non-routine-maintenance stuff that pops up
19:53:29 <zobel> you mean in rt.d.o or on irc?
19:53:36 <enrico> zobel: both
19:53:50 <weasel> hmm.
19:53:55 <weasel> I don't think we ever counted them.
19:54:12 <zobel> i would say 20 if i would need to guess
19:54:15 <zobel> but i cant say
19:54:24 <weasel> I'd guess the usual rate would be maybe half a dozen a day
19:54:33 <weasel> but peaks are much higher
19:54:47 <weasel> there are days where we have more than a dozen requests to update the debian.org zone,
19:55:04 <zobel> most of the mails we get are "please install build-dep" on $foo in $bar-chroot
19:55:05 <weasel> tho nowadays symoon usually does that himself :) - no need for us anymore
19:55:24 <zobel> that has been handed over to porters
19:55:29 <weasel> "porters"
19:55:43 <weasel> we have two or three people that help out with installing build dependencies on our porter machines
19:55:54 <weasel> but they aren't really the porters, are they?
19:56:33 <weasel> at least the feedback to our request for people to deal with these kind of requests, from the people who nominally sponsor an arch was next to nil
19:56:36 <weasel> .
19:56:59 <zobel> if porters are listening here, please speak up
19:57:05 <zobel> .
19:57:15 <MadameZou> < mehdi> QUESTION: how many times do you test your backups? :)
19:57:38 <weasel> almost never
19:57:54 <weasel> at least the file backups.
19:57:58 <mehdi> you deserve a special prize!
19:58:03 <mehdi> :)
19:58:09 <weasel> some postgres backups we got to test regularly in the last couple months,
19:58:26 <weasel> since the pg slave ate its data in snapshot's streaming setup
19:59:16 <weasel> there might have been the odd request to restore a single file or two, but I don't think it happened more than once or twice in the last year
19:59:19 <weasel> .
19:59:21 * zobel currently doing live DSA'ing here at DC11 :)
19:59:32 <weasel> zobel: tell them to make a ticket
19:59:44 <zobel> .
19:59:48 <MadameZou> < zigo> QUESTION: How do you organize duties to watch over servers, so that someone is there to fix in case of an issue? Do you have a time table or some kind of schedule where one should be held responsible?
19:59:57 <MadameZou> similar to enrico's one :)
20:00:00 <weasel> no, we don't.
20:00:07 <weasel> we are around or we aren't.
20:00:24 <zigo> So, it's like, the one that catches the issues first fixes it?
20:00:25 <zobel> most things aren't urgent.
20:00:28 <weasel> I suppose if stuff broke badly somebody would notice and phone us up if we weren't online
20:00:35 <zobel> zigo: yes
20:00:36 <zigo> Then what if 2 DSA are trying to fix the issue at the same time?
20:00:40 <weasel> but most things just aren't time critical, as zobel mentioned
20:01:17 <weasel> usually you notice that it's already fixed and ask on IRC if somebody did it or if it fixed itself
20:01:20 <zobel> most things can be solved by teams beside DSA
20:01:31 <zobel> we are only needed if hardware breaks or so.
20:01:32 <zobel> .
20:01:46 <weasel> and then we have nagios that reports some issues,
20:01:55 <weasel> and it complains to irc, so we might claim something right there
20:02:03 <weasel> but usually it's just not a problem
20:02:04 <weasel> .
20:02:05 <MadameZou> < morphic> QUESTION: How is your interaction with other OS projects teams, like FreeBSD? Is there some common developers, common code, etc?
20:02:28 <weasel> not really, no.
20:02:39 <zobel> morphic: we are in contact with some other OSs, but not that much
20:02:46 <weasel> I know of two other projects that use userdir-ldap, but that doesn't really count, since it's the same people as in DSA :)
20:02:58 <weasel> (three)
20:03:11 <weasel> (four)
20:03:28 <MadameZou> nobody expects the spanish inquisition!
20:03:33 <weasel> indeed :)
20:03:39 <zobel> .
20:03:47 <weasel> and as mentioned at the beginning, dsa isn't that much about coding/developing stuff
20:04:03 <weasel> it's more about keeping stuff running and setting new machines up.
20:04:21 <weasel> so there isn't too much where interaction with other projects would be useful
20:04:22 <weasel> .
20:04:25 <zobel> morphic: we have contact to some other SA like from ubuntu or so.
20:04:38 <zobel> SA as in Sys Admins
20:04:47 <zobel> .
20:04:49 <weasel> and there's the oss-infra list run by the osuosl people,
20:04:50 <MadameZou> < mehdi> QUESTION: are _all_ debian.org machines running Debian?
20:04:51 <weasel> but that's mostly idle
20:04:56 <weasel> yes.
20:05:08 <weasel> tho ubuntu LTS sounds tempting :)
20:05:13 <weasel> (and the stupid firmware policy sucks)
20:05:31 <weasel> (with squeeze most of our machines got non-free added to their sources.list)
20:06:03 <weasel> .
20:06:18 <Black_Prince> I wanted to ask if there was some non-free/proprietary software running on debian machines, but you already answered :)
20:06:30 <zobel> weasel: did sibelius just lost its pg again? :)
20:06:31 <weasel> kernel stuff, raid stuff, health stuff
20:06:54 <weasel> zobel: sibelius never lost its pg.  and stabile didn't right now.
20:07:05 <MadameZou> < mehdi> QUESTION: would it make sense to have a machine on which you test your backups?
20:07:23 <zobel> mh
20:07:28 <weasel> since we don't have full backups but only some service related directories, I'm not sure.
20:07:32 <weasel> what would you test?
20:08:07 <zobel> mehdi: what do you want to have tested?
20:08:08 <mehdi> that the backups work and are useful
20:08:15 <mehdi> (sorry, network laggish)
20:08:52 <mehdi> otherwise, it's "Suprise!" each time you need them
20:08:59 <zobel> but yes, we should test desaster recovery for db.d.o at one point.
20:08:59 <mehdi> *Surprise, even
20:09:19 <weasel> well, restoring a machine would probably be a few hours or work since we don't have full/disaster recovery backups.
20:09:28 <weasel> we only back up /etc/ and some /srv directories.
20:09:44 <weasel> so if stuff breaks it's re-set up and pick out the data you need from backup
20:09:45 <zobel> .
20:09:56 <weasel> that's probably a bit of work
20:10:24 <zobel> done
20:10:27 <weasel> I'm more worried about not having a backup of some service
20:10:33 <weasel> than the backup being broken
20:10:50 <weasel> requesting stuff be backed up is each team's job (i.e. they have to tell us to back up /srv/$whatever)
20:10:53 <weasel> and they might not have done that.
20:10:54 <weasel> .
20:10:55 <mehdi> if all services are backuped, but backups are broken... :)
20:11:11 <mehdi> but, yes, you answered my question, thanks!
20:11:16 <MadameZou> < babilen> QUESTION: apt-get, aptitude, wajig, synaptic or software center?
20:11:17 <weasel> then it's still only debian and we didn't lose any money.
20:11:24 <weasel> apt-get.
20:11:24 <weasel> .
20:11:28 <zobel> apt-get
20:11:29 <zobel> .
20:11:36 <MadameZou> < zigo> QUESTION: Since you mentioned that you have many places where machines are hosted, how do you keep track of all contacts for each machine? Do you store that in a Git or something, so that all DSA can share the info?
20:11:37 <zobel> or puppet letting do it!
20:12:00 <weasel> some stuff is stored implicitly in the group config of our password management tool
20:12:18 <weasel> other stuff is hidden somewhere in the debian-admin list mailbox or in memory
20:12:41 <weasel> zobel: any other places?
20:12:55 <weasel> oh right, ldap sponsor: field
20:12:56 <zobel> not that i am aware of... your brain? :)
20:13:10 <weasel> .
20:13:11 <zobel> but we already had that.
20:13:12 <zobel> .
20:13:28 <MadameZou> < babilen> QUESTION: Is Debian infrastructure frequently targetted in attacks or are we flying below the radar? What measures do you take to secure systems?
20:13:43 <weasel> I could tell you, but ... :)
20:13:55 <weasel> we are not aware of any specific attacks, but maybe we just don't notice.
20:14:43 <weasel> keep systems current, do those kernel update reboots.  hope that our users protect their systems as well
20:14:57 <zobel> babilen: we run some sort on IDS
20:15:07 <weasel> some basic firewalling, but unfortunately nothing too fancy
20:15:11 <zobel> to notice changes.
20:15:21 <babilen> "some sort of" (don't answer if you don't want to)
20:15:21 <weasel> people need to ssh in form all over the world.  that makes it hard to lock things down
20:15:28 <weasel> samhain
20:15:41 <weasel> (which is a pain because it doesn't work too well on things that aren't amd64/i386)
20:16:04 <weasel> and it spams us with a million mails (literally) whenver we do point release upgrades
20:16:14 <babilen> heh
20:16:20 <MadameZou> < alex_muntada> QUESTION: how do you plan reboots?
20:16:22 <MadameZou> ops
20:16:31 <weasel> we don't.  I just do them.
20:16:32 <MadameZou> sorry, this is the next one :)
20:16:35 <babilen> np
20:16:39 <babilen> .
20:16:57 <weasel> it depends on the machine however, how we do it
20:17:07 <weasel> for porter machines, when nobody is logged in, just shutdown -r now.
20:17:19 <zobel> alex_muntada: we try to not reboot machines that serve the same service
20:17:32 <weasel> for security/web/etc mirrorors (stuff that is dns hosted by geo[123].d.o), we shutdown -r 30 or something like that,
20:17:47 <weasel> a script picks up there is a shutdown running, and derotated the host from DNS
20:17:54 <weasel> once it's back, it gets added to dns again.
20:18:11 <alex_muntada> very interesting, thanks
20:18:15 <zobel> for buildds, we wait for the build to finish
20:18:16 <weasel> for buildds, we run buildd-reboot, which tells buildd to stop (after this build) and then reboots when buildd has shut down
20:18:37 <weasel> for services that aren't redundant (planet, manpages, etc) we just reboot them
20:18:50 <weasel> ideally all user targeted services would be redundant,
20:18:53 <zobel> ha, i see a local admin joining in here...
20:18:56 <zobel> hi paravoid!
20:19:23 <weasel> but unfortuantely not all are
20:19:24 <weasel> .
20:19:54 <zobel> MadameZou: maybe paravoid can tell a bit how he works together with DSA?
20:19:55 <weasel> (there's nothing DSA can do about that if the team doesn't want to)
20:20:07 <MadameZou> < mehdi> QUESTION: what's the difference between local admin and dsa? (maybe this one?)
20:20:35 <zobel> mehdi: local admins do only have (if at all) root access to their machines.
20:20:38 <zobel> DSA has to all
20:20:53 <mehdi> define "their" machines
20:21:06 <zobel> the machines hosted at their data center.
20:21:34 <zobel> mehdi: does that answer your question?
20:21:36 <weasel> and they aren't supposed to change anything or use it for much, unless we ask them nicely
20:21:45 <mehdi> zobel: yes, thanks
20:22:00 <zobel> mehdi: you can direct that question to paravoid if you want :)
20:22:03 <weasel> local admin are the people we talk to when we need hardware replaced or there are some networking issues
20:22:06 <weasel> .
20:22:26 <zobel> .
20:22:49 <MadameZou> < mehdi> QUESTION: do you have local admins next to each server?
20:22:54 <MadameZou> last question ^
20:22:58 <MadameZou> :D
20:23:18 <zobel> i am not sure how many local admins run the machines below their desk
20:23:25 <weasel> for varying values of local
20:23:32 <zobel> but i don't think we have that.
20:23:57 <zobel> we have some local admins working in the same building though.
20:24:07 <zobel> so they can work over if needed.
20:24:07 <weasel> I think it used to be that every d.o machine had a DD local admin next to it
20:24:26 <weasel> but these people moved on or graduated, and nowadays more machine are at nice data centers,
20:24:40 <weasel> so sometimes the 'local admin' is support@data-center
20:24:51 <mehdi> can we say "s/DD local admin/local DD/"?
20:25:06 <weasel> hm?
20:25:13 <mehdi> (don't know how much the difference is important:))
20:25:27 <mehdi> or maybe it's the same
20:25:30 <weasel> I'm not sure what you're trying to say
20:25:42 <mehdi> nevermind, I got my answer :)
20:25:45 <zobel> in most cases we had DDs introducing machines to us in their datacenter.
20:25:46 <weasel> not all people who touch our hardware are DDs nowadays.
20:25:53 <zobel> if that is what mehdi wants to know.
20:26:02 <weasel> for instance franck we just shipped to brown.edu,
20:26:11 <weasel> and they racked it for us and cabled it and all
20:26:18 <weasel> not a DD near it
20:26:24 <paravoid> I was to leave grnet though, it's more likely that a generic support@ would replace me rather than a local DD.
20:26:25 <zobel> same for villa and lobos
20:26:34 <mehdi> okay
20:26:48 <weasel> zobel: well, for villa and lobos the hardware came with the hosting I think
20:26:55 <paravoid> i.e. the admins are more tied to the facility hosting the machines, rather than Debian
20:26:58 <zobel> yes
20:27:00 <weasel> franck, we bought and had HP ship to brown
20:27:09 <weasel> yup
20:27:19 <zobel> any more questions?
20:27:26 <zobel> or are we don?
20:27:29 <zobel> done!
20:27:40 <MadameZou> so, thanks zobel and weasel for this nice session. And thanks to all attendees for their question.
20:27:55 <weasel> thanks
20:27:55 <MadameZou> s/question/questions/
20:27:56 <mehdi> \o/
20:28:08 <alex_muntada> zobel, weasel, MadameZou: congrats, great session!
20:28:09 <MadameZou> \o/ mehdi is our questions guy :)
20:28:14 * weasel goes back to minting bitcoins on all our machines
20:28:17 <MadameZou> alex_muntada: thank you for joining
20:28:25 <morphic> DSA. nice job, thank you :)
20:28:34 <MadameZou> and thank you zobel and weasel (and sgran) for your great work on debian machines
20:28:47 <weasel> you're welcome.  it's fun.  sometimes
20:28:56 <zobel> thanks for listening to us
20:28:59 <MadameZou> #endmeeting