Do not use "test" as the name of an executable test file.
test is a shell built-in.
References:
/usr/share/doc/package
Unix / Programming
Information
Many long info documents can be obtained as paperbacks from GNU.
The next four sections contain sample scripts in different languages for
creating a text file of account information to be added to
/etc/passwd using a batch processor such as the
newusers program. Each script requires as input a file with lines
of the form first_name last_name password. (Actual user home
directories will not be created via these scripts.)
Reading shell scripts is the best way to understand how a
Unix-like system works. Here, I give some pointers and reminders for shell
programming. See Shell
Mistakes to learn from mistakes.
References for Bash:
bash(1)
BASH Programming -
Introduction HOWTO as starter information.
(Install the bash-doc package to see the example files.)
Short program example (creates account entries for newusers from
standard input):
#!/bin/bash
# (C) Osamu Aoki Sun Aug 26 16:53:55 UTC 2001 Public Domain
pid=1000;
while read n1 n2 n3 ; do
if [ ${n1:0:1} != "#" ]; then
let pid=$pid+1
echo ${n1}_${n2}:password:${pid}:${pid}:,,,/home/${n1}_${n2}:/bin/bash
fi
done
Several packages provide a POSIX shell in Debian:
dash (Sarge)
ash (Woody)
bash
pdksh
If you are writing a shell script for portability, it is best to write it as a
POSIX shell script. Use /bin/sh linked to ash (or
dash) to test its POSIX compliance. Avoid writing scripts with
"bashisms" or the "zshisms". For example, avoid:
The description for the shell in this document applies only for the POSIX type
shells and thus does not apply for the csh type shells including
tcsh.
Several special parameters to remember:
$0 = name of the shell or shell script
$1 = first(1) shell argument
...
$9 = ninth(9) shell argument
$# = number of positional parameters
"$*" = "$1 $2 $3 $4 ... $n"
"$@" = "$1" "$2" "$3" "$4" ... "$n"
$? = exit status of the most recent command
$$ = PID of this shell script
$! = PID of most recently started background job
Basic parameter expansions to remember:
Form If var is set If var is not set
${var:-string} $var string
${var:+string} string null
${var:=string} $var string
(and run var=string)
${var:?string} $var (echo string and then exit)
Here, the colon `:' in all of these operators is actually optional.
Basic parameter substitutions to remember:
Form Result
${var%suffix} Remove smallest suffix pattern
${var%%suffix} Remove largest suffix pattern
${var#prefix} Remove smallest prefix pattern
${var##prefix} Remove largest prefix pattern
Basic redirection to remember (here the [n] is an optional number to specify the file descriptor):
[n]> file Redirect stdout (or n) to file.
[n]>> file Append stdout (or n) to file.
[n]< file Redirect stdin (or n) from file.
[n1]>&n2 Redirect stdout (or n1) to n2.
2> file >&2 Redirect stdout and stderr to file.
> file 2>&1 Redirect stdout and stderr to file.
| command Pipe stdout to command.
2>&1 | command Pipe stderr and stdout to command.
Here,
The shell allows you to open files using the exec built-in with an
arbitrary file descriptor.
$ echo Hello >foo
$ exec 3<foo 4>bar # open files
$ cat <&3 >&4 # redirect stdin to 3, stdout to 4
$ exec 3<&- 4>&- # close files
$ cat bar
Hello
Here n<&- and n>&- mean to close the file descriptor n.
Each command returns an exit status which can be used for conditional expressions:
Note that the use here of a 0 value to mean "true" differs from the
usual convention in some other areas of computing. Also, `[' is the equivalent
of the test command, which evaluates its arguments up to `]' as a
conditional expression.
Basic conditional idioms to remember are:
command && if_success_run_this_command_too || true
command || if_not_success_run_this_command_instead
if [ conditional_expression ]; then
if_success_run_this_command
else
if_not_success_run_this_command
fi
Here || true was needed to ensure this shell script will not exit at this line accidentally when shell is invoked with -e flag.
File comparison operators in the conditional expression are:
-e file True if file exists.
-d file True if file exists and is a directory.
-f file True if file exists and is a regular file.
-w file True if file exists and is writable.
-x file True if file exists and is executable.
file1 -nt file2 True if file1 is newer than file2. (modification)
file1 -ot file2 True if file1 is older than file2. (modification)
file1 -ef file2 True if they are the same device and inode numbers.
String comparison operators in the conditional expression are:
-z str True if the length of str is zero.
-n str True if the length of str is non-zero.
str1 == str2 True if the strings are equal.
str1 = str2 True if the strings are equal.
("=" should be used in place of "==" for strict POSIX compliance)
str1 != str2 True if the strings are not equal.
str1 < str2 True if str1 sorts before str2 (locale dependent).
str1 > str2 True if str1 sorts after str2 (locale dependent).
Arithmetic integer comparison operators in the conditional expression are -eq, -ne, -lt, -le, -gt, and -ge.
The shell processes a script as follows:
Single quotes within double quotes have no effect.
Executing set -x in the shell or invoking the shell with -x option make the shell to print all of commands executed. This is quite handy for debugging.
References for Awk:
mawk(1) and gawk(1)
Short program example (creates newusers command entry):
#!/usr/bin/awk -f
# Script to create a file suitable for use in the 'newusers' command,
# from a file consisting of user IDs and passwords in the form:
# first_name last_name password
# Copyright (c) KMSelf Sat Aug 25 20:47:38 PDT 2001
# Distributed under GNU GPL v 2, or at your option, any later version.
# This program is distributed WITHOUT ANY WARRANTY.
BEGIN {
# Assign starting UID, GID
if ( ARGC > 2 ) {
startuid = ARGV[1]
delete ARGV[1]
}
else {
printf( "Usage: newusers startUID file\n" \
" where:\n" \
" startUID is the starting userid to add, and\n" \
" file is an input file in form:\n" \
" first_name last_name password\n" \
)
exit
}
infile = ARGV[1]
printf( "Starting UID: %s\n\n", startuid )
}
/^#/ { next }
{
++record
first = $1
last = $2
passwd = $3
user= substr( tolower( first ), 1, 1 ) tolower( last )
uid = startuid + record - 1
gid = uid
printf( "%s:%s:%d:%d:%s %s,,/home/%s:/bin/bash\n", \
user, passwd, uid, gid, first, last, user \
)
}
Two packages provide POSIX awk in Debian:
mawk
gawk
This is the interpreter on a Unix-like system.
References for Perl:
perl(1)
The Perl Directory
Short program example (creates newusers command entry):
#!/usr/bin/perl
# (C) Osamu Aoki Sun Aug 26 16:53:55 UTC 2001 Public Domain
$pid=1000;
while (<STDIN>) {
if (/^#/) { next;}
chop;
$pid++;
($n1, $n2, $n3) = split / /;
print $n1,"_",$n2,":", $n3, ":",$pid,
":",$pid,",,,/home/",$n1,"_",$n2,":/bin/bash\n"
}
Install Perl module module_name:
# perl -MCPAN -e 'install module_name'
It's a nice object-oriented interpreter.
References for Python:
python(1)
Python Programming Language
Short program example (creates newusers command entry):
#! /usr/bin/env python
import sys, string
# (C) Osamu Aoki Sun Aug 26 16:53:55 UTC 2001 Public Domain
# Ported from awk script by KMSelf Sat Aug 25 20:47:38 PDT 2001
# This program is distributed WITHOUT ANY WARRANTY.
def usages():
print \
"Usage: ", sys.argv[0], " start_UID [filename]\n" \
"\tstartUID is the starting userid to add.\n" \
"\tfilename is input filename. If not specified, standard input.\n\n" \
"Input file format:\n"\
"\tfirst_name last_name password\n"
return 1
def parsefile(startuid):
#
# main filtering
#
uid = startuid
while 1:
line = infile.readline()
if not line:
break
if line[0] == '#':
continue
(first, last, passwd) = string.split(string.lower(line))
# above crashes with wrong # of parameters :-)
user = first[0] + last
gid = uid
lineout = "%s:%s:%d:%d:%s %s,,/home/%s:/bin/bash\n" % \
(user, passwd, uid, gid, first, last, user)
sys.stdout.write(lineout)
+uid
if __name__ == '__main__':
if len(sys.argv) == 1:
usages()
else:
uid = int(sys.argv[1])
#print "# UID start from: %d\n" % uid
if len(sys.argv) > 1:
infilename = string.join(sys.argv[2:])
infile = open(infilename, 'r')
#print "# Read file from: %s\n\n" % infilename
else:
infile = sys.stdin
parsefile(uid)
References for Make:
make(1)
Simple automatic variables:
Rule syntax:
target: [ prerequisites ... ]
[TAB] command1
[TAB] -command2 # ignore errors
[TAB] @command3 # suppress echoing
Here [TAB] is a TAB code. Each line is interpreted by the shell
after make variable substitution. Use \ at the end
of a line to continue the script. Use $$ to enter $
for environment values for a shell script.
Implicit rules for the target and prerequisites can be written, for example, as:
%: %.c header.h
or,
%.o: %.c header.h
Here, the target contains the character % (exactly one of them). The % can match any nonempty substring in the actual target filenames. The prerequisites likewise use % to show how their names relate to the actual target name.
Suffix rules are the obsolete way of defining
implicit rules for make. They are still supported in GNU
make for compatibility, but use equivalent pattern rules whenever
possible:
old suffix rule --> new pattern rule
.c: --> % : %.c
.c.o: --> %.o: %.c
Automatic variables for the rule:
foo.o: new1.c new2.c old1.c new3.c
$@ == foo.o (target)
$< == new1.c (first one)
$? == new1.c new2.c new3.c (newer ones)
$^ == new1.c new2.c old1.c new3.c (all)
$* == `%' matched stem in the target pattern.
Variable references:
foo1 := bar # One-time expansion
foo2 = bar # Recursive expansion
foo3 += bar # Append
SRCS := $(wildcard *.c)
OBJS := $(foo:c=o)
OBJS := $(foo:%.c=%.o)
OBJS := $(patsubst %.c,%.o,$(foo))
DIRS = $(dir directory/filename.ext) # Extracts "directory"
$(notdir NAMES...), $(basename NAMES...), $(suffix NAMES...) ...
Run make -p -f/dev/null to see automatic internal rules.
Preparation:
# apt-get install glibc-doc manpages-dev libc6-dev gcc
References for C:
gcc(1)
each_C_library_function_name(3)
gcc)
A simple example to compile example.c with a library
libm into an executable run_example:
$ cat > example.c << EOF
#include <stdio.h>
#include <math.h>
#include <string.h>
int main(int argc, char **argv, char **envp){
double x;
char y[11];
x=sqrt(argc+7.5);
strncpy(y, argv[0], 10); /* prevent buffer overflow */
y[10] = '\0'; /* fill to make sure string ends with '\0' */
printf("%5i, %5.3f, %10s, %10s\n", argc, x, y, argv[1]);
return 0;
}
EOF
$ gcc -Wall -g -o run_example example.c -lm
$ ./run_example
1, 2.915, ./run_exam, (null)
$ ./run_example 1234567890qwerty
2, 3.082, ./run_exam, 1234567890qwerty
Here, -lm is needed to link library
libm for sqrt(). The actual library
is in /lib/ with filename libm.so.6, which is a
symlink to libm-2.1.3.so.
Look at the last parameter in the output text. There are more than 10 characters even though %10s is specified.
The use of pointer memory operation functions without boundary checks, such as
sprintf and strcpy, is deprecated to prevent buffer
overflow exploits that leverage the above overrun effects. Instead, use
snprintf and strncpy.
gdbPreparation:
# apt-get install gdb
References for gdb:
gdb(1)
http://www.unknownroad.com/rtfm/gdbtut/gdbtoc.html
Use gdb to debug a program compiled with the -g
option. Many commands can be abbreviated. Tab expansion works as in the
shell.
$ gdb program
(gdb) b 1 # set breakpoint at line 1
(gdb) run arg1 arg2 arg3 # run program
(gdb) next # next line
...
(gdb) step # step forward
...
(gdb) p parm # print parm
...
(gdb) p parm=12 # set value to 12
For debugging from within Emacs, refer to Editor command summary (Emacs, Vim), Section 11.3.4.
Since all installed binaries should be stripped on the Debian system by
default, most debugging symbols are removed. In order to make gdb
useful for debugging Debian packages, pertinent packages need to be rebuild
with following care:
debian/control to bump the package version.
See Policy
10.1 for more info.
Use ldd to find out a program's dependency on libraries:
$ ldd /bin/ls
librt.so.1 => /lib/librt.so.1 (0x4001e000)
libc.so.6 => /lib/libc.so.6 (0x40030000)
libpthread.so.0 => /lib/libpthread.so.0 (0x40153000)
/lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)
For ls to work in a chrooted environment, the above
libraries must be available in your chrooted environment.
The following commands will also be useful:
strace: trace system calls and signals
ltrace: trace library calls
There are several memory leak detection tools available in Debian.
njamd
valgrind
dmalloc
electric-fence
memprof
memwatch (not packaged, get this from memwatch.)
mpatrol
leaktracer
libgc6
Parasoft. (non-free, commercial for
fee)
Also check out Debugging
Tools for Dynamic Storage Allocation and Memory Management.
flex is a fast lexical analyzer generator.
References for flex:
flex(1)
You need to provide your own main() and yywrap(), or
your program.l should look like this to compile without a library
(yywrap is a macro; %option main turns on
%option noyywrap implicitly):
%option main
%%
.|\n ECHO ;
%%
Alternatively, you may compile with the -lfl linker option at the
end of your cc command line (like AT&T-Lex with
-ll). No %option is needed in this case.
Several packages provide a Yacc-compatible LALR parser generator in Debian:
bison: GNU LALR parser generator
byacc: The Berkeley LALR parser generator
byyacc: Backtracking parser generator based on byacc
References for bison:
bison(1)
You need to provide your own main() and yyerror().
main() calls yyparse() which calls
yylex(), usually created with FleX.
%%
%%
autoconf is a tool for producing shell scripts that automatically
configure software source code packages to adapt to many kinds of UNIX-like
systems using the entire GNU build system.
autoconf produces the configuration script configure.
configure automatically creates a customized Makefile
using the Makefile.in template.
Debian does not touch files in /usr/local/ (see Supporting diversity, Section 2.5). So
if you compile a program from source, install it into /usr/local/
so it will not interfere with Debian.
$ cd src
$ ./configure --prefix=/usr/local
$ make
$ make install # this puts the files in the system
If you still have the source and if it uses
autoconf/automake and if you can remember how you
configured it:
$ ./configure all-of-the-options-you-gave-it
# make uninstall
Alternatively, if you are absolutely sure that the install process puts files
only under /usr/local/ and there is nothing important there, you
can erase all its contents by:
# find /usr/local -type f -print0 | xargs -0 rm -f
If you are not sure where files are installed, you should consider using
checkinstall, which provides a clean path for the uninstall.
Basic interactive dynamic web pages can be made as follows:
program.*) on the web server will receive
decoded parameters "VAR1=VAL1 VAR2=VAL2 VAR3=VAL3" as
the contents of environment variable "QUERY_STRING" and
executes itself.
For security reasons it is better not to hand craft new hacks for parsing CGI
parameters. There are established modules for them in Perl (see Perl, Section 13.4) and Python (see Python, Section 13.5). PHP comes with these functionality. When
client data storage is needed, cookies are used. When client side data
processing is needed, javascript is frequently used.
For more, see The Common
Gateway Interface, The Apache
Software Foundation, and JavaScript.
Searching "CGI tutorial" on Google by typing encoded URL http://www.google.com/search?hl=en&ie=UTF-8&q=CGI+tutorial directly to the browser address is a good way to see the CGI script in action on the Google server.
Traditionally, roff is the main Unix text processing system.
See roff(7), groff(7), groff(1),
grotty(1), troff(1), groff_mdoc(7),
groff_man(7), groff_ms(7), groff_me(7),
groff_mm(7), and info groff.
A good tutorial on -me macros exists. If you have
groff (1.18 or newer), find
/usr/share/doc/groff/meintro.me.gz and do the following:
$ zcat /usr/share/doc/groff/meintro.me.gz | \
groff -Tascii -me - | less -R
The following will make a completely plain text file:
$ zcat /usr/share/doc/groff/meintro.me.gz | \
GROFF_NO_SGR=1 groff -Tascii -me - | col -b -x > meintro.txt
For printing, use PostScript output.
$ groff -Tps meintro.txt | lpr
$ groff -Tps meintro.txt | mpage -2 | lpr
Preparation:
# apt-get install debiandoc-sgml debiandoc-sgml-doc
References for debiandoc-sgml:
/usr/share/doc/debiandoc-sgml-doc
debiandoc-sgml(1)
DocBook:
The Definitive Guide, by Walsh and Muellner, (O'Reilly)
(package docbook-defguide)
SGML enables management of multiple formats of a document. One easy SGML system is Debiandoc, which is used here. This requires minor conversion from original text files for the following characters:
To mark a section as a nonprintable comment, enter:
<!-- State issue here ... -->
To mark a section with a switchable comment, enter:
<![ %FIXME; [ State issue here ... ]]>
In SGML, the first definition of an entity wins. For example:
<!entity % qref "INCLUDE">
<![ %qref; [ <!entity param "Data 1"> ]]>
<!entity param "Data 2">
¶m;
This ends up as "Data 1". If the first line has "IGNORE" instead of "INCLUDE", this ends up as "Data 2" (the second line is a conditional statement). Also, repeating phrases can be defined in advance separately from the context.
<!entity whoisthis "my">
Hello &whoisthis; friend.
This is &whoisthis; book.
This results in the following:
Hello my friend.
This is my book.
See the short SGML example sample.sgml in the examples.
When SGML documents become bigger, sometimes TeX which is used as the backend text processor may cause errors. See TeX/LaTeX, Section 13.9.3.
Preparation:
# tasksel # select Miscellaneous --> TeX/LaTeX environment
References for LaTeX:
The teTeX HOWTO: The
Linux-teTeX Local Guide
tex(1)
latex(1)
This is the most powerful typesetting environment. Many SGML processors use
this as their back end text processor. Lyx provided by lyx,
lyx-xforms, or lyx-qt and GNU TeXmacs provided by
texmacs package offers nice WYSIWYG editing environment for LaTeX
while many use Emacs and Vim as the choice for the source editor.
There are many online resources available:
teTeX - A Documentation
Guide (tetex-doc package)
A Quick Introduction
to LaTeX
A Simple
Guide to Latex/Lyx
Word
Processing Using LaTeX
Local
User Guide to teTeX/LaTeX
When documents become bigger, sometimes TeX may cause errors. You must
increase pool size in /etc/texmf/texmf.cnf (or more appropriately
edit /etc/texmf/texmf.d/95NonPath and run
update-texmf) to fix this.
Instead of writing code containing documentation, the literate programmer writes documentation containing code. This approach ensures a good documentation for a program.
For more on literate-programming, see Literate Programming.
Preparation:
# apt-get install nowebm
References for Noweb:
This is a WEB-like literate-programming tool which is simpler while providing
extensibility and language-independence. [72] When noweb is invoked, it writes the program
source code to the output files mentioned in the noweb file, and it writes a
TeX file for typeset documentation.
The Debian ifupdown package is a fine example.
$ apt-get source ifupdown
$ cd ifupdown*
$ make ifupdown.pdf ifupdown.ps
Preparation:
# apt-get install doxygen doxygen-doc doxygen-gui
References for Doxygen (created by doxygen!):
It can generate HTML, RTF, Unix manual pages, PostScript, and PDF (using LaTeX)
documentation for C++, C, Java, IDL, and to some extent PHP and C# programs.
Doxygen is compatible to JavaDoc (1.1), Qt-Doc, KDOC and was specifically
designed to be used for projects that make use of Troll Tech's Qt toolkit. It creates include
dependency graphs, collaboration diagrams, and graphical class hierarchy graphs
even for not documented programs. The output is similar to Qt's documentation.
Preparation:
# apt-get install debian-policy developers-reference \
maint-guide dh-make debhelper
# apt-get install packaging-manual # if Potato
References for packaging:
dh-make(1)
Quick-and-dirty method to Package a single binary per Joey Hess.
# mkdir -p mypkg/usr/bin mypkg/DEBIAN
# cp binary mypkg/usr/bin
# cat > mypkg/DEBIAN/control
Package: mypackage
Version: 1
Architecture: i386
Maintainer: Joey Hess <joeyh@debian.org>
Description: my little package
Don't expect much.
^D
# dpkg-deb -b mypkg
Use dh_make from the dh-make package to create a
baseline package. Then, proceed according to instructions in
dh-make(1). This uses debhelper in
debian/rules.
An older approach is to use deb-make from the debmake
package. This uses no debhelper scripts and depends only on the
shell.
For examples of multiple-source packages, see "mc" (dpkg-source
-x mc_4.5.54.dsc), which uses "sys-build.mk" by Adam Heath
(doogie@debian.org), and
"glibc" (dpkg-source -x glibc_2.2.4-1.dsc), which uses
another system by the late Joel Klecker (espy@debian.org).
Debian Reference
CVS, Mon Apr 3 22:57:03 UTC 2005osamu@debian.orgdsewell@virginia.edu