System Administration

with the

Python Standard Library

John Pinner

www.linuxemporium.co.uk

Where I'm Coming from

• I've been using Unix for 30 years
  • (had a brief flirtation with Windows)
• Always using Bourne shell for 'sysadmin' scripting
  • latterly bash
• 'Serious' programming
  • machine code
  • assembler
  • BASIC yuk!
  • 4GL yuk!
  • FORTRAN errrmmm!
  • Pascal
  • C
  • Python

Where I am Now

• Using Python and C for Development
  • Python apps
  • C libraries


• Still using bash scripting for sysadmin
  • force of habit
• What is scripting?
  • hacks which you throw away?

Why do I want to Change?

(my scripting habits)

• We wrote a payroll
  • PayThyme
  • GPL'd
  • on Linux
• Using
  • C for the calculation engine
  • Python/PyQt for the UI and reports
  • shelling out for
    • backups, etc
    • through laziness
    • we were running on Linux, after all

Why do I want to Change?

• Linux with Python made this easy


• Python has a comprehensive Standard Library
  • comes 'with batteries included'
• The Python os module
  • Python's interface to the platform it's running on
  • has a 'system' command
    • just like C - system() (3)
• On Unix this gives the Python programmer a lazy option
  • Example

Why Should I Change?

• We decided to port PayThyme to Windows


• Python, C and Qt are multi-platform
  • so what's the problem?


• The lazy options came home to roost
  • luckily not too many of them


• All the places where we'd 'shelled out'
  • using Linux system-specific commands
  • had to be recoded
  • decided to do it portably

How did we make PayThyme multi-platform?

• Entirely with Python


• Three main problems
  • menuing system used spawn
    • Unix/Linux specific
  • used os.system() for
    • backups using Linux tar
    • simple file operations (copy, chmod, etc )
  • packaging
    • a whole different subject
    • later ?

Enter the Python Standard Library

• sys module
  • the interface to the Python Runtime Environment
  • allows us to view and manipulate the RTE
• sys.path - the Python module search path
  • a list, which you can change
• sys.exit() - exit function
  • in fact, it raises an exception
• sys.modules - dictionary of loaded modules
• sys.getrefcount(object) - returns reference count of object
• sys.setprofile() - set profiler function
  • - called when a function is entered and exited
• sys.stdin, stdout, stderr
  • - standard io streams, use Python file interface conventions
• sys.platform - what operating system we're running on

os Module

• The interface to the operating system
  • Provides operating system services
  • Some of this is system-dependent
• os.name - name of os
  • - 'posix', 'nt', 'os2', 'mac', 'ce' or 'riscos'
• os.path - set to path module needed by resident os
  • - posixpath, ntpath or macpath
• os.curdir - current directory

os Module

• os.sep - pathname separator
  • - for portability use this in os.system(cmd)
• os.linesep - end of line separator
  • - '\n', '\r\n' or `\r'
• os.environ - dictionary of environment variables
• os.getenv(varname) - get value of an environment variable
• os stat(path) - performs a stat system call on a file

Process Handling

• os.popen() - open a pipe
  • - eg printer = os.popen( '/usr/bin/lpr' )
  • - also popen2, popen3, popen4


• os.system(cmd) - runs operating system command cmd


• os has functions from the native environment
  • eg posix - spawnl, spawnv, etc
  • threading may be more portable

threading module Summary

• class Thread(name=None,target=None, args=(),kwargs={}):
  • name : optional name for thread
  • target : t.run() calls target with arguments args, kwargs


• Thread methods
  • t.start() : makes t active, and executes t.run() in a thread
  • t.isAlive() : returns True if thread is alive
  • t.join(timeout=None) : suspends execution until t terminates


• Example

Making PayThyme cross-platform

• Menuing system
  • used threading module instead of spawn


• 'Shelling out'
  • File operations - used shutil module
  • provides Unix-like file copy, mode-setting etc
    • Example
  • Backups - used tarfile module

Conclusion

• Using Python could give us cross-platform sysadmin capabilities


• The same scripts could be used on Unix Linux and Windows


• Backups could be restored cross-platform

Prize - portability across platforms

• How to find the time to do it?


• It's comfortable to carry on with the old ways
  • we're a Linux shop
  • reluctance to change
  • familiarity

How to force the issue?

• propose a talk for the ACCU

Making the Change

• The problem - missing find
• Find
  • prime example of the difference between Windows and Unix


• Windows
  • very easy to use
  • nice GUI interface
  • anyone can use it
• Unix
  • command line interface
  • very powerful
    • lots of options, plays well with other utilities
  • arcane, difficult to master
    • not for the end user

First: Walking a file system

• From Python Cookbook


• walk.py

Second: find

• Now we can walk the file system we can find files


• Requirements
  • match file names by pattern
  • exclude file names by pattern
  • include files 'newer than' some file
    • eg for incremental backups

Second: find - how?

• From Standard Library:
  • re module
  • ConfigParser
• re
  • Regular Expression matching, similar to Perl regex's
  • Simple Example
• ConfigParser - parses Windows-type config files
  • each section contains name,value pairs
  • supports Python format strings using section namespace
• The rest we do ourselves
  • find.py

Third: Editing Files

• We've already found the file
• Requirements:
  • take the list of files we've found
  • replace all occurrences of 'find_string' with 'replace_string'
  • make a list of files which are candidates for changing
  • optionally make the changes
• How?
  • standard Python file and string handling
  • find_and_edit.py

Fourth: Making the Changes

• Whoops!
  • maybe we should back them up first!

Fourth: Taking a backup

• Requirements:
  • Take the list of files to be changes
  • Back them up


• How ?
  • Python Standard Library - tarfile module
  • backup_first.py

Fifth: Making the Changes

• Run the edit function with the update option


• find_and_replace.py

Sixth: Make it more Useful

• Let's provide command line options


• How ?
  • argparse module
  • not Python Standard Library
    • which has getopt and optparse
    • getopt - like C getopt and GNU getopt()
    • optparse - more Pythonic, object-oriented

Python argparse module

• Even more Pythonic than optparse
• Developed by Steven Bethard
  • inspired by optparse
• Easy to use
  • good help and documention facilities
  • type-checking of arguments
• Not in Python Standard Library
  • available from the Python Package Index - PyPI
    • aka The Cheese Shop
• Same basic functions as optparse
  • Example arg8.py

some argparse features

• Allows choices
  • Example arg9.py


• Processes all commandline arguments
  • positional as well as options
  • positional arguments are not just left hanging


• Has built in file handling
  • saves coding
  • Example arg10.py

some argparse features

• You can define your own type checking
  • Example arg11.py


• No callback feature as in optparse
  • but you could use the type checking feature
  • Example arg12.py


• Advanced features
  • sub-parsers, user-defined Actions

Sixth: Make it more Useful

• with argparse
• use the functions we've developed in a module
• write a program
  • imports the module
    • renamed pyadmin
  • handles commandline arguments with argparse
• freplace

Maintainability and Reliability

• Test
  • if __name__ == '__main__' :
    • your test code
  • unittest - Standard Library test harness
  • Example
• Check routinely
  • pychecker
  • pylint
  • PyFlakes

The End

• Questions ?