System Administration
with the
Python Standard Library
John Pinner
www.linuxemporium.co.uk
Where I'm Coming from
-
I've been using Unix for 30 years
-
(had a brief flirtation with Windows)
-
Always using Bourne shell for 'sysadmin' scripting
-
'Serious' programming
-
machine code
-
assembler
-
BASIC
yuk!
-
4GL
yuk!
-
FORTRAN
errrmmm!
-
Pascal
-
C
-
Python

Where I am Now
-
Using Python and C for Development
-
Still using bash scripting for sysadmin
-
What is scripting?
-
hacks which you throw away?
Why do I want to Change?
(my scripting habits)
-
We wrote a payroll
-
Using
-
C for the calculation engine
-
Python/PyQt for the UI and reports
-
shelling out for
-
backups, etc
-
through laziness
-
we were running on Linux, after all
Why do I want to Change?
-
Linux with Python made this easy
-
Python has a comprehensive Standard Library
-
comes 'with batteries included'
-
The Python os module
-
Python's interface to the platform it's running on
-
has a 'system' command
-
just like C - system() (3)
-
On Unix this gives the Python programmer a lazy option
Why Should I Change?
-
We decided to port PayThyme to Windows
-
Python, C and Qt are multi-platform
-
The lazy options came home to roost
-
luckily not too many of them
-
All the places where we'd 'shelled out'
-
using Linux system-specific commands
-
had to be recoded
-
decided to do it portably
How did we make PayThyme multi-platform?
-
Entirely with Python
-
Three main problems
-
menuing system used spawn
-
used os.system() for
-
backups using Linux tar
-
simple file operations (copy, chmod, etc )
-
packaging
-
a whole different subject
-
later ?
Enter the Python Standard Library
-
sys module
-
the interface to the Python Runtime Environment
-
allows us to view and manipulate the RTE
-
sys.path - the Python module search path
-
a list, which you can change
-
sys.exit() - exit function
-
in fact, it raises an exception
-
sys.modules - dictionary of loaded modules
-
sys.getrefcount(object) - returns reference count of object
-
sys.setprofile() - set profiler function
-
- called when a function is entered and exited
-
sys.stdin, stdout, stderr
-
- standard io streams, use Python file interface conventions
-
sys.platform - what operating system we're running on
os Module
-
The interface to the operating system
-
Provides operating system services
-
Some of this is system-dependent
-
os.name - name of os
-
- 'posix', 'nt', 'os2', 'mac', 'ce' or 'riscos'
-
os.path - set to path module needed by resident os
-
- posixpath, ntpath or macpath
-
os.curdir - current directory
os Module
-
os.sep - pathname separator
-
- for portability use this in os.system(cmd)
-
os.linesep - end of line separator
-
os.environ - dictionary of environment variables
-
os.getenv(varname) - get value of an environment variable
-
os stat(path) - performs a stat system call on a file
Process Handling
-
os.popen() - open a pipe
-
- eg printer = os.popen( '/usr/bin/lpr' )
-
- also popen2, popen3, popen4
-
os.system(cmd) - runs operating system command cmd
-
os has functions from the native environment
-
eg posix - spawnl, spawnv, etc
-
threading may be more portable
threading module Summary
-
class Thread(name=None,target=None, args=(),kwargs={}):
-
name : optional name for thread
-
target : t.run() calls target with arguments args, kwargs
-
Thread methods
-
t.start() : makes t active, and executes t.run() in a thread
-
t.isAlive() : returns True if thread is alive
-
t.join(timeout=None) : suspends execution until t terminates
-
Example
Making PayThyme cross-platform
-
Menuing system
-
used threading module instead of spawn
-
'Shelling out'
-
File operations - used shutil module
-
provides Unix-like file copy, mode-setting etc
-
Backups - used tarfile module
Conclusion
-
Using Python could give us cross-platform sysadmin capabilities
-
The same scripts could be used on Unix Linux and Windows
-
Backups could be restored cross-platform
Prize - portability across platforms
-
How to find the time to do it?
-
It's comfortable to carry on with the old ways
-
we're a Linux shop
-
reluctance to change
-
familiarity
How to force the issue?
-
propose a talk for the ACCU
Making the Change
-
The problem - missing find
-
Find
-
prime example of the difference between Windows and Unix
-
Windows
-
very easy to use
-
nice GUI interface
-
anyone can use it
-
Unix
-
command line interface
-
very powerful
-
lots of options, plays well with other utilities
-
arcane, difficult to master
First: Walking a file system
Second: find
-
Now we can walk the file system we can find files
-
Requirements
-
match file names by pattern
-
exclude file names by pattern
-
include files 'newer than' some file
-
eg for incremental backups
Second: find - how?
-
From Standard Library:
-
re
-
ConfigParser - parses Windows-type config files
-
each section contains name,value pairs
-
supports Python format strings using section namespace
-
The rest we do ourselves
Third: Editing Files
-
We've already found the file
-
Requirements:
-
take the list of files we've found
-
replace all occurrences of 'find_string' with 'replace_string'
-
make a list of files which are candidates for changing
-
optionally make the changes
-
How?
Fourth: Making the Changes
-
Whoops!
-
maybe we should back them up first!
Fourth: Taking a backup
-
Requirements:
-
Take the list of files to be changes
-
Back them up
-
How ?
Fifth: Making the Changes
Sixth: Make it more Useful
-
Let's provide command line options
-
How ?
-
argparse module
-
not Python Standard Library
-
which has getopt and optparse
-
getopt - like C getopt and GNU getopt()
-
optparse - more Pythonic, object-oriented
Python argparse module
-
Even more Pythonic than optparse
-
Developed by Steven Bethard
-
Easy to use
-
good help and documention facilities
-
type-checking of arguments
-
Not in Python Standard Library
-
available from the Python Package Index - PyPI
-
Same basic functions as optparse
some argparse features
-
Allows choices
-
Processes all commandline arguments
-
positional as well as options
-
positional arguments are not just left hanging
-
Has built in file handling
some argparse features
-
You can define your own type checking
-
No callback feature as in optparse
-
but you could use the type checking feature
-
Example arg12.py
-
Advanced features
-
sub-parsers, user-defined Actions
Sixth: Make it more Useful
-
with argparse
-
use the functions we've developed in a module
-
write a program
-
imports the module
-
handles commandline arguments with argparse
-
freplace
Maintainability and Reliability
-
Test
-
if __name__ == '__main__' :
-
unittest - Standard Library test harness
-
Example
-
Check routinely
-
pychecker
-
pylint
-
PyFlakes