20:02:23 <MadameZou> #startmeeting
20:02:23 <MeetBot> Meeting started Thu Dec  2 20:02:23 2010 UTC.  The chair is MadameZou. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:02:23 <MeetBot> Useful Commands: #action #agreed #help #info #idea #link #topic.
20:02:28 <POX> Hi! My name is Piotr Ożarowski and I will try to explain few basic things about packaging Python modules and applications written in Python.
20:02:40 <POX> Please ask questions at any time on this channel. If possible, try to wait for an answer to previous question or wait for questions & answers part. TIA
20:02:58 <POX> I hope to finish the first part (with 2 simple examples) fast so that we'll have more time for Q&A.
20:03:24 <POX> Please `apt-get install devscripts python-all-dev python-stdeb`, we'll need them (you're doing development work in up to date Sid, right?).
20:03:33 <POX> *** POLICY ***
20:03:42 <POX> First, few notes about Debian Python Policy: (AKA boring stuff that you have to know ;)
20:04:11 <POX> * Python interpreter in Debian (since Python 2.6) is modified to use /usr/lib/pythonX.Y/dist-packages instead of site-packages
20:04:24 <POX> this way we can avoid conflicts with manually installed interpreter
20:04:50 <POX> It will also use /usr/local as --prefix by default (everyone who had to uninstall something ez_install installed (and overwrote our files), knows how important this change is),
20:05:12 <POX> you can pass --install-layout=deb to setup.py's install to set all the Debian's defaults (f.e. --prefix=/usr)
20:05:21 <POX> * we have few special packages:
20:05:29 <POX> - python (which depends on default Python version, currently python2.6; this package also provides f.e. dh_python2 helper)
20:05:49 <POX> - python-dev (like python, but additionally depends on pythonX.Y-dev package which provides header files used while building extensions)
20:06:11 <POX> - python-dbg (depends on debug version of interpreter)
20:06:21 <POX> - python-all (which depends on all supported Python versions, python2.5 and python2.6 right now, +python2.7 in python-all from experimental)
20:06:39 <POX> - python-all-dev (like python-all, but depends on -dev packages as well)
20:06:58 <POX> - python-all-dbg (depends on all supported debug interpreters)
20:06:59 <POX> see python-defaults source package for more info
20:07:34 <POX> * packages with Python modules should support all Python versions supported by Debian (the ones python-all installs)
20:07:54 <POX> this can be done by creating symlinks to Python pure modules (.py files) at install time (dh_pycentral and dh_pysupport can take care of that for you)
20:08:01 <POX> or by shipping all symlinks inside the .deb file (dh_python2 does that)
20:08:16 <POX> If a package provides public Python extension(s) (.so files), it has to ship them for all supported Python versions
20:08:37 <POX> * binary packages with public modules should follow python-modulename naming schema (python-namespace.modulename for namespace packages, f.e. python-pyside.qtgui).
20:08:57 <POX> If documentation is big (bigger than the module itself), you can consider adding python-foo-doc binary package.
20:09:10 <POX> I usually ship Sphinx generated docs in /usr/share/doc/python-foo-doc/html directory with a /usr/share/doc/python-foo-doc/rst → /usr/share/doc/python-foo/html/_sources symlink
20:09:31 <POX> (and /usr/share/doc/python-foo/html → /usr/share/doc/python-foo-doc/html)
20:09:43 <POX> If you want to make the Python extension optional (when module doesn't require it) you can ship it in python-foo-ext binary package and Recommend this package in python-foo
20:09:54 <POX> * private modules should be installed in private directory such as /usr/share/packagename/ or /usr/lib/packagename/
20:10:15 <POX> * Python 2.X and Python 3.X should be treated as separate runtime systems
20:10:34 <POX> You still can use the same source package name, but all binary packages providing Python 3 modules should be prefixed with "python3-" (python3-mako, python3-lxml, etc.)
20:10:49 <POX> There are special packages for Python 3 as well (python3, python3-all, python3-all-dev, ...), python3 provides dh_python3 helper
20:11:25 <POX> Right now Python 3 modules/extensions are not built by dh so you have to modify your debian/rules file, we'll fix that before Wheezy release
20:11:46 <POX> (in CDBS v0.4.90 to support Python 3, you'll have to add "include /usr/share/cdbs/1/class/python3-distutils.mk" line to debian/rules and python3-foo package(s) to debian/control)
20:12:05 <POX> dh already invokes dh_python3 when you add "--with python3" option, but it doesn't build modules/extensions for Python 3 yet
20:12:24 <POX> let me know if it's too fast, BTW
20:12:49 <POX> * Python 3 installs all public modules and extensions to /usr/lib/python3/dist-packages (all Python 3 interpreters share the same directory!)
20:12:52 <go8ose> POX: the pace is good for me
20:13:09 <POX> any questions about the policy?
20:14:27 <POX> ok, few things you should do *before* packaging:
20:14:38 <POX> * check if the package already exists or current maintainer wants someone else to take over (O, RFA bugs against wnpp package),
20:14:53 <POX> someone requested for package (RFP) or intends to package it (ITP) - contact ITP submitter if it's been a while and bug is still not closed.
20:15:11 <POX> Remember to check modulename, python-modulename, pydistname, python-pydistname source package names
20:15:57 <POX> pydistname as in what upstream uses for to name the package (PasteDeploy for paste.deploy module, etc.)
20:16:12 <POX> (it's not guaranteed that someone will spot a duplicate after reading your ITP)
20:16:30 <POX> * download the tarball, check if its license matches DFSG (`licensecheck -r unpacked-tarball/` could help)
20:16:56 <POX> start with writing debian/copyright file to not waste time - it happened to me once or twice that after doing all the packaging work, I figured out I actually cannot upload it due to license issues
20:17:05 <POX> contact upstream author if you still want the package in Debian.
20:17:50 <POX> * check if it really is worth packaging; do we have alternatives in Debian? If so, will you be able to convince others that we should have yet another package doing XYZ?
20:18:12 <POX> * file ITP (with *final* long description of the package) or rename O/RFP bug
20:18:21 <POX> still no questions?
20:18:39 <POX> I'll follow with first example then
20:18:52 <POX> *** EXAMPLE 1 - Python extension ***
20:19:03 <POX> $ wget http://pypi.python.org/packages/source/M/MarkupSafe/MarkupSafe-0.11.tar.gz
20:19:28 <POX> $ py2dsc -m 'Piotr Ożarowski <piotr@debian.org>' MarkupSafe-0.11.tar.gz
20:19:39 <POX> py2dsc comes from python-stdeb package
20:19:56 <POX> -m is needed for now, as I didn't patch it yet to use DEBEMAIL
20:20:06 <POX> $ cd deb_dist/markupsafe-0.11
20:20:14 <POX> or if you're already in the directory with unpacked files:
20:20:20 <POX> $ python setup.py --command-packages=stdeb.command debianize
20:20:35 <POX> this will create debian dir in the current directory
20:20:48 <POX> copy the previously prepared copyright file or prepare it now (again, this can save your time)
20:21:01 <POX> ok, lets take a look at debian directory
20:22:02 <POX> hmm... there's a debian/patches subdirectory, but we didn't create patches yet...
20:22:09 * POX is opening debian/patches/debian-changes-0.11-1
20:22:17 <POX> ah, right, distutils is not cleaning the .egg-info directory
20:22:29 <POX> lets remove this patch and clean the directory via dh_clean (distutils will regenerate it)
20:22:38 <POX> $ quilt delete debian/patches/debian-changes-0.11-1
20:22:44 <POX> $ rm -rf debian/patches
20:22:59 <POX> (we don't need this directory, at least for now)
20:23:08 <POX> $ echo 'MarkupSafe.egg-info/*' > debian/clean
20:23:17 <POX> this should stop dpkg-source from adding this patch again
20:23:50 <POX> (dh_clean doesn't remove directories, but removing files inside is enough for us)
20:25:13 <POX> debian/changelog...
20:25:20 <POX> we already have ITP bug number so lets replace the "source package ..." line with "Initial release. Closes: 123456" (where 123456 is our ITP number)
20:25:34 <POX> now lets try to build the package for the first time...
20:25:41 <POX> $ debuild
20:25:52 <POX> $ lintian -i ../markupsafe_0.11-1_*.changes
20:26:00 <POX> it worked! no errors and warnings, looks like packaging Python extensions is not that hard after all :-)
20:26:13 <POX> you can invoke lintian with -IE if you want more issues to be reported, but it's enough to have the package clean of errors (E:) and warnings (W:)
20:26:35 <POX> please note that `lintian *changes` will check source package and binary packages, there's no need to invoke` lintian *dev separately`
20:26:40 <POX> *deb, sorry
20:27:00 <POX> ok, now lets try to install all binary packages generated from our source package
20:27:11 <POX> $ sudo debi ../markupsafe_0.11-1_*.changes
20:27:26 <POX> now we can test the package
20:28:06 <POX> there's a standard interface to test Python modules (python setup.py test) but it will test files in current directory
20:28:19 <POX> remember to change the current directory as Python tries to search for modules in .
20:28:38 <POX> I forget about it every second week ;)
20:29:24 <POX> you'll find information about how to test the module in README
20:29:31 <POX> or other upstream docs
20:29:45 <POX> Notes about new upstream releases:
20:29:52 <POX> * when new upstream version will be available, don't use py2dsc again, just invoke `uupdate path/to/new-tarball` in directory with old version
20:30:27 <POX> it will create ../python-foo-newversion/ directory and use dch to add new entry in debian/changelog
20:30:48 <POX> dch is a nice tool, BTW (man dch)
20:30:59 <POX> (`apt-get source yourpackage` will download it for you if don't have it anymore).
20:31:20 <POX> (old sources)
20:31:28 <POX> * always skim through `debdiff foo_oldversion.dsc foo_newversion.dsc` output
20:31:40 <POX> hint: `debdiff ... | filterdiff -x '*/doc/*' -x '*/po/*'` will ignore changes in docs and translations (remember to adjust dir names)
20:32:00 <POX> you can find changes in debian directory that you forgot to document in debian/changelog
20:32:07 <POX> or changes in upstream code that require bumping minimum build/runtime versions or are dangerous and should be patched/reported upstream
20:32:48 <POX> still no question? I guess you are waiting for the Q&A part, ok
20:33:07 <POX> *** EXAMPLE 2 - Python application ***
20:33:14 <POX> first, lets download the tarball
20:33:21 <POX> $ wget http://downloads.sourceforge.net/project/ljcharm/charm/charm-1.9.1/charm-1.9.1.tar.gz
20:33:34 <POX> unpack it and create the debian directory the same way you create it for all other packages
20:33:45 <POX> you can do it with py2dsc as well, but we'll need to make few adjustments later:
20:33:52 <POX> $ py2dsc -m 'Piotr Ożarowski <piotr@debian.org>' charm-1.9.1.tar.gz
20:34:02 <POX> $ cd deb_dist/charm-1.9.1
20:34:11 <POX> open debian/control file and...
20:34:18 <POX> - rename binary package name to "charm"
20:34:25 <POX> we will not provide public module, so "python-" prefix is not needed
20:34:36 <POX> - remove "Breaks: ${python:Breaks}" line as well, for the same reason
20:34:47 <POX> - replace python-all with python in Build-Depends, we don't need all interpreters
20:34:57 <POX> - remove "--buildsystem=python_distutils" from debian/rules, there's no Makefile
20:35:10 <POX> README.charm mentions that this application needs Python 2.5 so tell dh_python2 that by adding "X-Python-Version: >= 2.5" in source stanza
20:35:45 <POX> (for Python 3 applications use X-Python3-Version and dh_python3
20:35:48 <POX> )
20:35:55 <POX> now the important part:
20:36:01 <POX> in debian/rules, add these three lines:
20:36:08 <POX> override_dh_auto_install:
20:36:21 <POX> python setup.py install --root=debian/charm --install-layout=deb --install-lib=/usr/share/charm --install-scripts=/usr/share/charm
20:36:26 <POX> override_dh_auto_build:
20:36:33 <POX> it will tell distutils that we want to use private directory
20:36:49 <POX> this directory is not in sys.path, so charm script is installed (via --install-scripts) to this directory as well
20:37:14 <POX> --root is distutils' DESTDIR
20:37:23 <POX> (rename the script if the module has exactly the same name, new name doesn't matter, we'll use the original one in /usr/bin anyway)
20:37:38 <POX> create debian/links file with "/usr/share/charm/charm /usr/bin/charm" inside
20:37:49 <POX> this file will be picked up by dh_link which will create a /usr/bin/charm symlink
20:38:25 <POX> use f.e. "/usr/share/charm/run /usr/bin/charm" if module name is "charm" as well
20:38:49 <POX> but remember to rename the binary in /usr/share/charm
20:39:03 <POX> we had to add override_dh_auto_build line because dh detected setup.py file and wants to build modules for all
20:39:12 <POX> ok, lets build and install the package
20:39:20 <POX> $ debuild
20:39:28 <POX> $ sudo debi ../charm_1.9.1-1_*.changes
20:39:41 <POX> and test it...
20:40:08 <POX> changing directory is not needed in this case, but I always open new terminal to do tests, just in case
20:40:14 <POX> $ charm
20:40:24 <POX> wow, it works :)
20:40:32 <umang> POX: What exactly does {$python:Breaks} expand to and why do we need to remove it?
20:40:44 <POX> there are warnings about hashlib module, but still
20:41:12 <POX> umang: it will move "python (>= first_supported), python (<< last_supported)" from Depends into Breaks: python (<< first_supported), python (>= last_supported)
20:41:31 <POX> it's used for Python modules, thet's why I added it to stdeb few days ago
20:42:06 <umang> OK. Thanks!
20:42:18 <POX> if you will not add Breaks, dh_python2 will add this dependency in Depends and then lintian will cry
20:42:57 <umang> So why do we need a different approach for modules and apps?
20:43:33 <POX> you mean why private modules are used in applications?
20:43:49 <POX> mainly to not pollute glogal namespace
20:43:54 <POX> global*
20:44:20 <umang> No. Why do we need the breaks for modules and not for applications? Is it because apps have private modules?
20:44:29 <go8ose> umang: I suspect you mean why different way of building. One reason I can think of (and I could be wrong) is that apps don't have to support all python versions
20:44:42 <POX> (two applications that provide "gui" module cannot be installed at the same time if both would use public namespace)
20:44:53 <POX> Breaks is simply not used in applications
20:45:06 <POX> you can leave it in debian/control, but it will do nothing
20:45:22 <umang> Ah. That makes sense.
20:45:34 <POX> private module can support only one Python version at the same time
20:45:53 <POX> dh_python2 will regenerate .pyc files if default python version will change]
20:46:28 <POX> well, it will change with Python 3 (which can support multiple Python versions even with private modules)
20:46:47 <POX> I guess we already started Q&A part
20:46:50 <POX> please ask
20:47:04 <go8ose> I missed the first packaging session. Is there a transcript available?
20:47:27 <POX> there's a link on http://wiki.debian.org/DebianWomen/TrainingSessions
20:47:49 <MadameZou> go8ose: you can find it on http://wiki.debian.org/IntroDebianPackaging
20:47:50 <umang> Sorry, just a clarification about my previous question. For modules we need to add Breaks because if we don't the module will compile on versions of python that it doesn't support? For apps, there are only private modules and they all use only one version at a time.
20:48:14 <POX> Breaks will just move some dependencies from Depends to Breaks
20:48:22 <POX> nothing else
20:48:40 <POX> (it's easier to override it if it's in Breaks)
20:49:25 <POX> f.e. the package is still usable with pythonX.Y, you can use it although it Breaks pythonX.Z
20:49:26 <umang> Thanks!
20:49:46 <go8ose> This isn't python specific, but it looks like the first tutorial didn't cover it. Is there a reference that describes what ./debian/rules tools like debuild (or dpkg-buildpackage, etc) call for building packages?
20:49:47 <POX> s/f.e./f.e. if/
20:50:46 <POX> debuild is actually calling dpkg-buildpackage... which is invoking debian/rules sometarget
20:50:54 <POX> it's described in Debian Policy
20:51:06 <POX> sometarget can be f.e. build
20:51:11 <POX> or binary-indep
20:51:40 <POX> `./debian/ruiles binary-indep` will build all architecture independent packages
20:51:53 <POX> `./debian/ruiles binary-arch` will build all architecture dependent packages
20:52:16 <POX> debuild is just a wrapper
20:52:19 <go8ose> Ah, I see, http://www.debian.org/doc/debian-policy/ch-source.html#s-debianrules
20:53:24 <POX> debian/rules is "just" a Makefile
20:55:30 <POX> dh_python2 tries to translate upstream's requires.txt into Debian's dependencies
20:55:43 <POX> but if upstream didn't provide this file
20:56:02 <POX> you can use tools like find_python_dependencies.py to figure out missing dependencies
20:56:09 <POX> http://svn.debian.org/viewsvn/python-modules/tools/
20:56:38 <go8ose> I often package internal python scripts and things for work, targetting the stable release. Is there a wiki page that summarises the 'state-of-the-art' for each release? Ie for lenny it might say you can use [[pycentral]] or [[dh_python]] or [[pysupport]]?
20:57:25 <POX> dh_python2 is introduced in Squeeze and in Wheezy I plan to drop python-central and pytnon-support
20:57:31 <POX> but that's just me :)
20:57:46 <POX> Matthias already agrees to drop python-central
20:58:15 <POX> for Python 3 we have only one helper (dh_python3)
20:58:44 <POX> I modified dh_pysupport and dh_pycentral to ignore python3-foo packages
20:58:54 <POX> dh_python3 ignores python-foo on the other hand
20:59:23 <POX> but you still have to tell debhelper which tool should handle private directories
20:59:51 <POX> i.e. all tools will try to bytecompile files in /usr/lib/foo/
21:01:45 <MadameZou> So, do you have questions for POX ?
21:02:14 <darkwise> thanks POX
21:02:36 <POX> any questions about dh_python2 or dh_python3?
21:02:42 <darkwise> POX: if we have some problem, how, where we can ask ?
21:02:55 <POX> I didn't write much doccumentation yet...
21:03:07 <POX> darkwise: #debian-python channel
21:03:18 <POX> or debian-python@list.debian.org
21:03:26 <POX> there are many smart guys there
21:03:37 <POX> lists*
21:03:42 <MadameZou> POX: can you explain us the procedure to join the python team in Debian?
21:04:03 <POX> http://wiki.debian.org/Teams/PythonModulesTeam/HowToJoin
21:04:28 <POX> you pick a package, start packaging it and contact us when you need help or an sponsor
21:04:28 <MadameZou> POX: thanks
21:05:11 <POX> we try to take a look at all DPMT / PAPT packages but it's mostly your responsibility to care about it
21:05:31 <aghisla> guys :)
21:05:42 <darkwise> POX: thanks again
21:05:52 <POX> you will also receive comments to your commits from time to time
21:06:07 <POX> mostly from me complaining about something ;)
21:06:18 <MadameZou> POX: *g*
21:06:53 <go8ose> POX: are there issues that make for common complaints from you? Do you see people repeatedly doing things incorrectly, that you want to mention here?
21:07:58 <POX> hmm.. the only common is about tagging before an upload (most packages are not ready before the first review)
21:09:32 <POX> ask on #debian-mentors or #debian-python (for Python related stuff)
21:09:37 <POX> we don't bite
21:10:22 <POX> well, if you'll go to debconf and tell me I'm your sponsor, I might bite (for all your mistakes from the past ;)
21:10:57 <MadameZou> POX: maybe is a little off topic but it seems we have not so many questions, so... How do you (as team) managed relationship with upstream and downstream?
21:11:02 * POX was just joking
21:11:48 <darkwise> :D
21:11:59 <POX> well, the only policy we have is: talk with upstream
21:12:04 <POX> introduce yourself'
21:12:13 <POX> forward her/him bugs
21:12:14 <POX> send patches
21:12:29 <POX> we also have quite some upstreams in the team
21:12:52 <POX> we try to help them a little bit more than we help others
21:14:25 <federico2> POX, I've never used pydeb, how would you compare it with dh_make?
21:14:48 <POX> s/pydeb/stdeb/?
21:14:53 <federico2> stdeb, sorry
21:15:01 <POX> well, dh_make doesn't know much about recent changes in Python policy
21:15:21 <POX> if any ;)
21:16:45 <valhalla> POX: do you have any suggestions for upstream authors? anything that they should do to help the packaging?
21:17:29 <POX> http://wiki.debian.org/UpstreamGuide
21:17:46 <POX> there's a Python section as well
21:18:16 <POX> there's also http://us.pycon.org/media/2010/talkdata/PyCon2010/038/paper.html
21:19:13 <POX> Stani (SPE and Phatch author) is one of out team members, BTW
21:19:46 <go8ose> POX: thanks for the tutorial
21:20:11 <POX> (he wrote "Cross-platform Application Development and Distribution" linked above with Nadia Alramli)
21:20:52 <POX> you can also ask Runa_ about Python stuff
21:21:14 <MadameZou> oh!
21:21:42 <MadameZou> so the python team is one of the women-friendly team ;)
21:21:59 <POX> we still didn't convince ana to join
21:22:04 * POX tried
21:22:36 <POX> ana sponsored many Python packages from me
21:22:57 <MadameZou> :)
21:24:09 <MadameZou> so, any other questions?
21:24:33 <POX> about sponsors: you can cheat a littile bit and check on this page: http://wiki.debian.org/SponsorChecklist what sponsors usually check
21:27:21 <federico2> POX, I'm comparing the debian/ directories build by py2dsc and dh_make for the same package
21:28:04 <POX> stdeb tries to parse some data from setup.py
21:28:07 <federico2> py2dsc is setting some fields in d/control to reasonable values, including the Buil-Depends
21:28:10 <POX> like package description
21:28:15 <federico2> it fetches some depends from setup.py
21:28:36 <POX> dh_make simply doesn't know much about distutils
21:28:37 <federico2> why is XS-Python-Version: current set?
21:28:48 <POX> and stdeb is... distutils' extension
21:29:25 <POX> written by Andrew Straw... who is DPMT member
21:29:58 <federico2> does  XS-Python-Version: current build for the default Python version or all the supported ones?
21:30:05 <POX> current is deprecated
21:30:27 <POX> it was used by pytohn-central to bytecompile for one Python version only
21:30:34 <federico2> uh, maybe I'm not using the latest version of stdeb
21:31:05 <POX> but since it's better to simply use private directories, we dropped "current" keyword
21:31:15 <federico2> I'm also curious about some directives in debian/rules:
21:31:30 <POX> I uploaded stdeb with few parches yesterday :)
21:31:31 <federico2> unexport CPPFLAGS and so on
21:31:34 <federico2> :D
21:31:43 <POX> (version dedicated to #debian-women ;)
21:32:00 <POX> patches*
21:32:21 <MadameZou> wow
21:32:24 <MadameZou> ;)
21:32:47 <POX> I also uploaded new python-defaults to experimental, with new features
21:32:53 <POX> like .pyinstall or .pyremove
21:33:11 <POX> to make pakcaging easier, but there were no questions about it
21:33:34 <POX> I mean, there were no problems that .pyinstall or .pyremove could solve
21:34:53 <POX> there are few more advanced issues mentioned in http://people.debian.org/~piotr/dc10_slides.pdf
21:35:29 <POX> I also recommend to subscribe debian-python@lists.debian.org, it's not that high traffic
21:36:06 <POX> f.e. pyinstall files are described here: http://lists.debian.org/debian-python/2010/11/msg00063.html
21:38:05 <POX> Packaging Python stuff is really not that scary, and it will be even easier with Python 3
21:41:26 <MadameZou> #endmeeting