20:02:23 #startmeeting 20:02:23 Meeting started Thu Dec 2 20:02:23 2010 UTC. The chair is MadameZou. Information about MeetBot at http://wiki.debian.org/MeetBot. 20:02:23 Useful Commands: #action #agreed #help #info #idea #link #topic. 20:02:28 Hi! My name is Piotr Ożarowski and I will try to explain few basic things about packaging Python modules and applications written in Python. 20:02:40 Please ask questions at any time on this channel. If possible, try to wait for an answer to previous question or wait for questions & answers part. TIA 20:02:58 I hope to finish the first part (with 2 simple examples) fast so that we'll have more time for Q&A. 20:03:24 Please `apt-get install devscripts python-all-dev python-stdeb`, we'll need them (you're doing development work in up to date Sid, right?). 20:03:33 *** POLICY *** 20:03:42 First, few notes about Debian Python Policy: (AKA boring stuff that you have to know ;) 20:04:11 * Python interpreter in Debian (since Python 2.6) is modified to use /usr/lib/pythonX.Y/dist-packages instead of site-packages 20:04:24 this way we can avoid conflicts with manually installed interpreter 20:04:50 It will also use /usr/local as --prefix by default (everyone who had to uninstall something ez_install installed (and overwrote our files), knows how important this change is), 20:05:12 you can pass --install-layout=deb to setup.py's install to set all the Debian's defaults (f.e. --prefix=/usr) 20:05:21 * we have few special packages: 20:05:29 - python (which depends on default Python version, currently python2.6; this package also provides f.e. dh_python2 helper) 20:05:49 - python-dev (like python, but additionally depends on pythonX.Y-dev package which provides header files used while building extensions) 20:06:11 - python-dbg (depends on debug version of interpreter) 20:06:21 - python-all (which depends on all supported Python versions, python2.5 and python2.6 right now, +python2.7 in python-all from experimental) 20:06:39 - python-all-dev (like python-all, but depends on -dev packages as well) 20:06:58 - python-all-dbg (depends on all supported debug interpreters) 20:06:59 see python-defaults source package for more info 20:07:34 * packages with Python modules should support all Python versions supported by Debian (the ones python-all installs) 20:07:54 this can be done by creating symlinks to Python pure modules (.py files) at install time (dh_pycentral and dh_pysupport can take care of that for you) 20:08:01 or by shipping all symlinks inside the .deb file (dh_python2 does that) 20:08:16 If a package provides public Python extension(s) (.so files), it has to ship them for all supported Python versions 20:08:37 * binary packages with public modules should follow python-modulename naming schema (python-namespace.modulename for namespace packages, f.e. python-pyside.qtgui). 20:08:57 If documentation is big (bigger than the module itself), you can consider adding python-foo-doc binary package. 20:09:10 I usually ship Sphinx generated docs in /usr/share/doc/python-foo-doc/html directory with a /usr/share/doc/python-foo-doc/rst → /usr/share/doc/python-foo/html/_sources symlink 20:09:31 (and /usr/share/doc/python-foo/html → /usr/share/doc/python-foo-doc/html) 20:09:43 If you want to make the Python extension optional (when module doesn't require it) you can ship it in python-foo-ext binary package and Recommend this package in python-foo 20:09:54 * private modules should be installed in private directory such as /usr/share/packagename/ or /usr/lib/packagename/ 20:10:15 * Python 2.X and Python 3.X should be treated as separate runtime systems 20:10:34 You still can use the same source package name, but all binary packages providing Python 3 modules should be prefixed with "python3-" (python3-mako, python3-lxml, etc.) 20:10:49 There are special packages for Python 3 as well (python3, python3-all, python3-all-dev, ...), python3 provides dh_python3 helper 20:11:25 Right now Python 3 modules/extensions are not built by dh so you have to modify your debian/rules file, we'll fix that before Wheezy release 20:11:46 (in CDBS v0.4.90 to support Python 3, you'll have to add "include /usr/share/cdbs/1/class/python3-distutils.mk" line to debian/rules and python3-foo package(s) to debian/control) 20:12:05 dh already invokes dh_python3 when you add "--with python3" option, but it doesn't build modules/extensions for Python 3 yet 20:12:24 let me know if it's too fast, BTW 20:12:49 * Python 3 installs all public modules and extensions to /usr/lib/python3/dist-packages (all Python 3 interpreters share the same directory!) 20:12:52 POX: the pace is good for me 20:13:09 any questions about the policy? 20:14:27 ok, few things you should do *before* packaging: 20:14:38 * check if the package already exists or current maintainer wants someone else to take over (O, RFA bugs against wnpp package), 20:14:53 someone requested for package (RFP) or intends to package it (ITP) - contact ITP submitter if it's been a while and bug is still not closed. 20:15:11 Remember to check modulename, python-modulename, pydistname, python-pydistname source package names 20:15:57 pydistname as in what upstream uses for to name the package (PasteDeploy for paste.deploy module, etc.) 20:16:12 (it's not guaranteed that someone will spot a duplicate after reading your ITP) 20:16:30 * download the tarball, check if its license matches DFSG (`licensecheck -r unpacked-tarball/` could help) 20:16:56 start with writing debian/copyright file to not waste time - it happened to me once or twice that after doing all the packaging work, I figured out I actually cannot upload it due to license issues 20:17:05 contact upstream author if you still want the package in Debian. 20:17:50 * check if it really is worth packaging; do we have alternatives in Debian? If so, will you be able to convince others that we should have yet another package doing XYZ? 20:18:12 * file ITP (with *final* long description of the package) or rename O/RFP bug 20:18:21 still no questions? 20:18:39 I'll follow with first example then 20:18:52 *** EXAMPLE 1 - Python extension *** 20:19:03 $ wget http://pypi.python.org/packages/source/M/MarkupSafe/MarkupSafe-0.11.tar.gz 20:19:28 $ py2dsc -m 'Piotr Ożarowski ' MarkupSafe-0.11.tar.gz 20:19:39 py2dsc comes from python-stdeb package 20:19:56 -m is needed for now, as I didn't patch it yet to use DEBEMAIL 20:20:06 $ cd deb_dist/markupsafe-0.11 20:20:14 or if you're already in the directory with unpacked files: 20:20:20 $ python setup.py --command-packages=stdeb.command debianize 20:20:35 this will create debian dir in the current directory 20:20:48 copy the previously prepared copyright file or prepare it now (again, this can save your time) 20:21:01 ok, lets take a look at debian directory 20:22:02 hmm... there's a debian/patches subdirectory, but we didn't create patches yet... 20:22:09 * POX is opening debian/patches/debian-changes-0.11-1 20:22:17 ah, right, distutils is not cleaning the .egg-info directory 20:22:29 lets remove this patch and clean the directory via dh_clean (distutils will regenerate it) 20:22:38 $ quilt delete debian/patches/debian-changes-0.11-1 20:22:44 $ rm -rf debian/patches 20:22:59 (we don't need this directory, at least for now) 20:23:08 $ echo 'MarkupSafe.egg-info/*' > debian/clean 20:23:17 this should stop dpkg-source from adding this patch again 20:23:50 (dh_clean doesn't remove directories, but removing files inside is enough for us) 20:25:13 debian/changelog... 20:25:20 we already have ITP bug number so lets replace the "source package ..." line with "Initial release. Closes: 123456" (where 123456 is our ITP number) 20:25:34 now lets try to build the package for the first time... 20:25:41 $ debuild 20:25:52 $ lintian -i ../markupsafe_0.11-1_*.changes 20:26:00 it worked! no errors and warnings, looks like packaging Python extensions is not that hard after all :-) 20:26:13 you can invoke lintian with -IE if you want more issues to be reported, but it's enough to have the package clean of errors (E:) and warnings (W:) 20:26:35 please note that `lintian *changes` will check source package and binary packages, there's no need to invoke` lintian *dev separately` 20:26:40 *deb, sorry 20:27:00 ok, now lets try to install all binary packages generated from our source package 20:27:11 $ sudo debi ../markupsafe_0.11-1_*.changes 20:27:26 now we can test the package 20:28:06 there's a standard interface to test Python modules (python setup.py test) but it will test files in current directory 20:28:19 remember to change the current directory as Python tries to search for modules in . 20:28:38 I forget about it every second week ;) 20:29:24 you'll find information about how to test the module in README 20:29:31 or other upstream docs 20:29:45 Notes about new upstream releases: 20:29:52 * when new upstream version will be available, don't use py2dsc again, just invoke `uupdate path/to/new-tarball` in directory with old version 20:30:27 it will create ../python-foo-newversion/ directory and use dch to add new entry in debian/changelog 20:30:48 dch is a nice tool, BTW (man dch) 20:30:59 (`apt-get source yourpackage` will download it for you if don't have it anymore). 20:31:20 (old sources) 20:31:28 * always skim through `debdiff foo_oldversion.dsc foo_newversion.dsc` output 20:31:40 hint: `debdiff ... | filterdiff -x '*/doc/*' -x '*/po/*'` will ignore changes in docs and translations (remember to adjust dir names) 20:32:00 you can find changes in debian directory that you forgot to document in debian/changelog 20:32:07 or changes in upstream code that require bumping minimum build/runtime versions or are dangerous and should be patched/reported upstream 20:32:48 still no question? I guess you are waiting for the Q&A part, ok 20:33:07 *** EXAMPLE 2 - Python application *** 20:33:14 first, lets download the tarball 20:33:21 $ wget http://downloads.sourceforge.net/project/ljcharm/charm/charm-1.9.1/charm-1.9.1.tar.gz 20:33:34 unpack it and create the debian directory the same way you create it for all other packages 20:33:45 you can do it with py2dsc as well, but we'll need to make few adjustments later: 20:33:52 $ py2dsc -m 'Piotr Ożarowski ' charm-1.9.1.tar.gz 20:34:02 $ cd deb_dist/charm-1.9.1 20:34:11 open debian/control file and... 20:34:18 - rename binary package name to "charm" 20:34:25 we will not provide public module, so "python-" prefix is not needed 20:34:36 - remove "Breaks: ${python:Breaks}" line as well, for the same reason 20:34:47 - replace python-all with python in Build-Depends, we don't need all interpreters 20:34:57 - remove "--buildsystem=python_distutils" from debian/rules, there's no Makefile 20:35:10 README.charm mentions that this application needs Python 2.5 so tell dh_python2 that by adding "X-Python-Version: >= 2.5" in source stanza 20:35:45 (for Python 3 applications use X-Python3-Version and dh_python3 20:35:48 ) 20:35:55 now the important part: 20:36:01 in debian/rules, add these three lines: 20:36:08 override_dh_auto_install: 20:36:21 python setup.py install --root=debian/charm --install-layout=deb --install-lib=/usr/share/charm --install-scripts=/usr/share/charm 20:36:26 override_dh_auto_build: 20:36:33 it will tell distutils that we want to use private directory 20:36:49 this directory is not in sys.path, so charm script is installed (via --install-scripts) to this directory as well 20:37:14 --root is distutils' DESTDIR 20:37:23 (rename the script if the module has exactly the same name, new name doesn't matter, we'll use the original one in /usr/bin anyway) 20:37:38 create debian/links file with "/usr/share/charm/charm /usr/bin/charm" inside 20:37:49 this file will be picked up by dh_link which will create a /usr/bin/charm symlink 20:38:25 use f.e. "/usr/share/charm/run /usr/bin/charm" if module name is "charm" as well 20:38:49 but remember to rename the binary in /usr/share/charm 20:39:03 we had to add override_dh_auto_build line because dh detected setup.py file and wants to build modules for all 20:39:12 ok, lets build and install the package 20:39:20 $ debuild 20:39:28 $ sudo debi ../charm_1.9.1-1_*.changes 20:39:41 and test it... 20:40:08 changing directory is not needed in this case, but I always open new terminal to do tests, just in case 20:40:14 $ charm 20:40:24 wow, it works :) 20:40:32 POX: What exactly does {$python:Breaks} expand to and why do we need to remove it? 20:40:44 there are warnings about hashlib module, but still 20:41:12 umang: it will move "python (>= first_supported), python (<< last_supported)" from Depends into Breaks: python (<< first_supported), python (>= last_supported) 20:41:31 it's used for Python modules, thet's why I added it to stdeb few days ago 20:42:06 OK. Thanks! 20:42:18 if you will not add Breaks, dh_python2 will add this dependency in Depends and then lintian will cry 20:42:57 So why do we need a different approach for modules and apps? 20:43:33 you mean why private modules are used in applications? 20:43:49 mainly to not pollute glogal namespace 20:43:54 global* 20:44:20 No. Why do we need the breaks for modules and not for applications? Is it because apps have private modules? 20:44:29 umang: I suspect you mean why different way of building. One reason I can think of (and I could be wrong) is that apps don't have to support all python versions 20:44:42 (two applications that provide "gui" module cannot be installed at the same time if both would use public namespace) 20:44:53 Breaks is simply not used in applications 20:45:06 you can leave it in debian/control, but it will do nothing 20:45:22 Ah. That makes sense. 20:45:34 private module can support only one Python version at the same time 20:45:53 dh_python2 will regenerate .pyc files if default python version will change] 20:46:28 well, it will change with Python 3 (which can support multiple Python versions even with private modules) 20:46:47 I guess we already started Q&A part 20:46:50 please ask 20:47:04 I missed the first packaging session. Is there a transcript available? 20:47:27 there's a link on http://wiki.debian.org/DebianWomen/TrainingSessions 20:47:49 go8ose: you can find it on http://wiki.debian.org/IntroDebianPackaging 20:47:50 Sorry, just a clarification about my previous question. For modules we need to add Breaks because if we don't the module will compile on versions of python that it doesn't support? For apps, there are only private modules and they all use only one version at a time. 20:48:14 Breaks will just move some dependencies from Depends to Breaks 20:48:22 nothing else 20:48:40 (it's easier to override it if it's in Breaks) 20:49:25 f.e. the package is still usable with pythonX.Y, you can use it although it Breaks pythonX.Z 20:49:26 Thanks! 20:49:46 This isn't python specific, but it looks like the first tutorial didn't cover it. Is there a reference that describes what ./debian/rules tools like debuild (or dpkg-buildpackage, etc) call for building packages? 20:49:47 s/f.e./f.e. if/ 20:50:46 debuild is actually calling dpkg-buildpackage... which is invoking debian/rules sometarget 20:50:54 it's described in Debian Policy 20:51:06 sometarget can be f.e. build 20:51:11 or binary-indep 20:51:40 `./debian/ruiles binary-indep` will build all architecture independent packages 20:51:53 `./debian/ruiles binary-arch` will build all architecture dependent packages 20:52:16 debuild is just a wrapper 20:52:19 Ah, I see, http://www.debian.org/doc/debian-policy/ch-source.html#s-debianrules 20:53:24 debian/rules is "just" a Makefile 20:55:30 dh_python2 tries to translate upstream's requires.txt into Debian's dependencies 20:55:43 but if upstream didn't provide this file 20:56:02 you can use tools like find_python_dependencies.py to figure out missing dependencies 20:56:09 http://svn.debian.org/viewsvn/python-modules/tools/ 20:56:38 I often package internal python scripts and things for work, targetting the stable release. Is there a wiki page that summarises the 'state-of-the-art' for each release? Ie for lenny it might say you can use [[pycentral]] or [[dh_python]] or [[pysupport]]? 20:57:25 dh_python2 is introduced in Squeeze and in Wheezy I plan to drop python-central and pytnon-support 20:57:31 but that's just me :) 20:57:46 Matthias already agrees to drop python-central 20:58:15 for Python 3 we have only one helper (dh_python3) 20:58:44 I modified dh_pysupport and dh_pycentral to ignore python3-foo packages 20:58:54 dh_python3 ignores python-foo on the other hand 20:59:23 but you still have to tell debhelper which tool should handle private directories 20:59:51 i.e. all tools will try to bytecompile files in /usr/lib/foo/ 21:01:45 So, do you have questions for POX ? 21:02:14 thanks POX 21:02:36 any questions about dh_python2 or dh_python3? 21:02:42 POX: if we have some problem, how, where we can ask ? 21:02:55 I didn't write much doccumentation yet... 21:03:07 darkwise: #debian-python channel 21:03:18 or debian-python@list.debian.org 21:03:26 there are many smart guys there 21:03:37 lists* 21:03:42 POX: can you explain us the procedure to join the python team in Debian? 21:04:03 http://wiki.debian.org/Teams/PythonModulesTeam/HowToJoin 21:04:28 you pick a package, start packaging it and contact us when you need help or an sponsor 21:04:28 POX: thanks 21:05:11 we try to take a look at all DPMT / PAPT packages but it's mostly your responsibility to care about it 21:05:31 guys :) 21:05:42 POX: thanks again 21:05:52 you will also receive comments to your commits from time to time 21:06:07 mostly from me complaining about something ;) 21:06:18 POX: *g* 21:06:53 POX: are there issues that make for common complaints from you? Do you see people repeatedly doing things incorrectly, that you want to mention here? 21:07:58 hmm.. the only common is about tagging before an upload (most packages are not ready before the first review) 21:09:32 ask on #debian-mentors or #debian-python (for Python related stuff) 21:09:37 we don't bite 21:10:22 well, if you'll go to debconf and tell me I'm your sponsor, I might bite (for all your mistakes from the past ;) 21:10:57 POX: maybe is a little off topic but it seems we have not so many questions, so... How do you (as team) managed relationship with upstream and downstream? 21:11:02 * POX was just joking 21:11:48 :D 21:11:59 well, the only policy we have is: talk with upstream 21:12:04 introduce yourself' 21:12:13 forward her/him bugs 21:12:14 send patches 21:12:29 we also have quite some upstreams in the team 21:12:52 we try to help them a little bit more than we help others 21:14:25 POX, I've never used pydeb, how would you compare it with dh_make? 21:14:48 s/pydeb/stdeb/? 21:14:53 stdeb, sorry 21:15:01 well, dh_make doesn't know much about recent changes in Python policy 21:15:21 if any ;) 21:16:45 POX: do you have any suggestions for upstream authors? anything that they should do to help the packaging? 21:17:29 http://wiki.debian.org/UpstreamGuide 21:17:46 there's a Python section as well 21:18:16 there's also http://us.pycon.org/media/2010/talkdata/PyCon2010/038/paper.html 21:19:13 Stani (SPE and Phatch author) is one of out team members, BTW 21:19:46 POX: thanks for the tutorial 21:20:11 (he wrote "Cross-platform Application Development and Distribution" linked above with Nadia Alramli) 21:20:52 you can also ask Runa_ about Python stuff 21:21:14 oh! 21:21:42 so the python team is one of the women-friendly team ;) 21:21:59 we still didn't convince ana to join 21:22:04 * POX tried 21:22:36 ana sponsored many Python packages from me 21:22:57 :) 21:24:09 so, any other questions? 21:24:33 about sponsors: you can cheat a littile bit and check on this page: http://wiki.debian.org/SponsorChecklist what sponsors usually check 21:27:21 POX, I'm comparing the debian/ directories build by py2dsc and dh_make for the same package 21:28:04 stdeb tries to parse some data from setup.py 21:28:07 py2dsc is setting some fields in d/control to reasonable values, including the Buil-Depends 21:28:10 like package description 21:28:15 it fetches some depends from setup.py 21:28:36 dh_make simply doesn't know much about distutils 21:28:37 why is XS-Python-Version: current set? 21:28:48 and stdeb is... distutils' extension 21:29:25 written by Andrew Straw... who is DPMT member 21:29:58 does XS-Python-Version: current build for the default Python version or all the supported ones? 21:30:05 current is deprecated 21:30:27 it was used by pytohn-central to bytecompile for one Python version only 21:30:34 uh, maybe I'm not using the latest version of stdeb 21:31:05 but since it's better to simply use private directories, we dropped "current" keyword 21:31:15 I'm also curious about some directives in debian/rules: 21:31:30 I uploaded stdeb with few parches yesterday :) 21:31:31 unexport CPPFLAGS and so on 21:31:34 :D 21:31:43 (version dedicated to #debian-women ;) 21:32:00 patches* 21:32:21 wow 21:32:24 ;) 21:32:47 I also uploaded new python-defaults to experimental, with new features 21:32:53 like .pyinstall or .pyremove 21:33:11 to make pakcaging easier, but there were no questions about it 21:33:34 I mean, there were no problems that .pyinstall or .pyremove could solve 21:34:53 there are few more advanced issues mentioned in http://people.debian.org/~piotr/dc10_slides.pdf 21:35:29 I also recommend to subscribe debian-python@lists.debian.org, it's not that high traffic 21:36:06 f.e. pyinstall files are described here: http://lists.debian.org/debian-python/2010/11/msg00063.html 21:38:05 Packaging Python stuff is really not that scary, and it will be even easier with Python 3 21:41:26 #endmeeting