=head1 NAME
perlebcdic - Considerations for running Perl on EBCDIC platforms
=head1 DESCRIPTION
An exploration of some of the issues facing Perl programmers
on EBCDIC based computers. We do not cover localization,
internationalization, or multi byte character set issues other
than some discussion of UTF-8 and UTF-EBCDIC.
Portions that are still incomplete are marked with XXX.
=head1 COMMON CHARACTER CODE SETS
=head2 ASCII
The American Standard Code for Information Interchange (ASCII or US-ASCII) is a
set of
integers running from 0 to 127 (decimal) that imply character
interpretation by the display and other systems of computers.
The range 0..127 can be covered by setting the bits in a 7-bit binary
digit, hence the set is sometimes referred to as a "7-bit ASCII".
ASCII was described by the American National Standards Institute
document ANSI X3.4-1986. It was also described by ISO 646:1991
(with localization for currency symbols). The full ASCII set is
given in the table below as the first 128 elements. Languages that
can be written adequately with the characters in ASCII include
English, Hawaiian, Indonesian, Swahili and some Native American
languages.
There are many character sets that extend the range of integers
from 0..2**7-1 up to 2**8-1, or 8 bit bytes (octets if you prefer).
One common one is the ISO 8859-1 character set.
=head2 ISO 8859
The ISO 8859-$n are a collection of character code sets from the
International Organization for Standardization (ISO) each of which
adds characters to the ASCII set that are typically found in European
languages many of which are based on the Roman, or Latin, alphabet.
=head2 Latin 1 (ISO 8859-1)
A particular 8-bit extension to ASCII that includes grave and acute
accented Latin characters. Languages that can employ ISO 8859-1
include all the languages covered by ASCII as well as Afrikaans,
Albanian, Basque, Catalan, Danish, Faroese, Finnish, Norwegian,
Portuguese, Spanish, and Swedish. Dutch is covered albeit without
the ij ligature. French is covered too but without the oe ligature.
German can use ISO 8859-1 but must do so without German-style
quotation marks. This set is based on Western European extensions
to ASCII and is commonly encountered in world wide web work.
In IBM character code set identification terminology ISO 8859-1 is
also known as CCSID 819 (or sometimes 0819 or even 00819).
=head2 EBCDIC
The Extended Binary Coded Decimal Interchange Code refers to a
large collection of slightly different single and multi byte
coded character sets that are different from ASCII or ISO 8859-1
and typically run on host computers. The EBCDIC encodings derive
from 8 bit byte extensions of Hollerith punched card encodings.
The layout on the cards was such that high bits were set for the
upper and lower case alphabet characters [a-z] and [A-Z], but there
were gaps within each Latin alphabet range.
Some IBM EBCDIC character sets may be known by character code set
identification numbers (CCSID numbers) or code page numbers. Leading
zero digits in CCSID numbers within this document are insignificant.
E.g. CCSID 0037 may be referred to as 37 in places.
Perl can be compiled on platforms that run any of three commonly used EBCDIC
character sets, listed below.
=head2 The 13 variant characters
Among IBM EBCDIC character code sets there are 13 characters that
are often mapped to different integer values. Those characters
are known as the 13 "variant" characters and are:
\ [ ] { } ^ ~ ! # | $ @ `
When Perl is compiled for a platform, it looks at some of these characters to
guess which EBCDIC character set the platform uses, and adapts itself
accordingly to that platform. If the platform uses a character set that is not
one of the three Perl knows about, Perl will either fail to compile, or
mistakenly and silently choose one of the three.
They are:
=head2 0037
Character code set ID 0037 is a mapping of the ASCII plus Latin-1
characters (i.e. ISO 8859-1) to an EBCDIC set. 0037 is used
in North American English locales on the OS/400 operating system
that runs on AS/400 computers. CCSID 37 differs from ISO 8859-1
in 237 places, in other words they agree on only 19 code point values.
=head2 1047
Character code set ID 1047 is also a mapping of the ASCII plus
Latin-1 characters (i.e. ISO 8859-1) to an EBCDIC set. 1047 is
used under Unix System Services for OS/390 or z/OS, and OpenEdition
for VM/ESA. CCSID 1047 differs from CCSID 0037 in eight places.
=head2 POSIX-BC
The EBCDIC code page in use on Siemens' BS2000 system is distinct from
1047 and 0037. It is identified below as the POSIX-BC set.
=head2 Unicode code points versus EBCDIC code points
In Unicode terminology a I is the number assigned to a
character: for example, in EBCDIC the character "A" is usually assigned
the number 193. In Unicode the character "A" is assigned the number 65.
This causes a problem with the semantics of the pack/unpack "U", which
are supposed to pack Unicode code points to characters and back to numbers.
The problem is: which code points to use for code points less than 256?
(for 256 and over there's no problem: Unicode code points are used)
In EBCDIC, for the low 256 the EBCDIC code points are used. This
means that the equivalences
pack("U", ord($character)) eq $character
unpack("U", $character) == ord $character
will hold. (If Unicode code points were applied consistently over
all the possible code points, pack("U",ord("A")) would in EBCDIC
equal I or chr(101), and unpack("U", "A") would equal
65, or I, not 193, or ord "A".)
=head2 Remaining Perl Unicode problems in EBCDIC
=over 4
=item *
Many of the remaining problems seem to be related to case-insensitive matching
=item *
The extensions Unicode::Collate and Unicode::Normalized are not
supported under EBCDIC, likewise for the encoding pragma.
=back
=head2 Unicode and UTF
UTF stands for C.
UTF-8 is an encoding of Unicode into a sequence of 8-bit byte chunks, based on
ASCII and Latin-1.
The length of a sequence required to represent a Unicode code point
depends on the ordinal number of that code point,
with larger numbers requiring more bytes.
UTF-EBCDIC is like UTF-8, but based on EBCDIC.
You may see the term C character or code point.
This simply means that the character has the same numeric
value when encoded as when not.
(Note that this is a very different concept from L
mentioned above.)
For example, the ordinal value of 'A' is 193 in most EBCDIC code pages,
and also is 193 when encoded in UTF-EBCDIC.
All other code points occupy at least two bytes when encoded.
In UTF-8, the code points corresponding to the lowest 128
ordinal numbers (0 - 127: the ASCII characters) are invariant.
In UTF-EBCDIC, there are 160 invariant characters.
(If you care, the EBCDIC invariants are those characters
which have ASCII equivalents, plus those that correspond to
the C1 controls (80..9f on ASCII platforms).)
A string encoded in UTF-EBCDIC may be longer (but never shorter) than
one encoded in UTF-8.
=head2 Using Encode
Starting from Perl 5.8 you can use the standard new module Encode
to translate from EBCDIC to Latin-1 code points.
Encode knows about more EBCDIC character sets than Perl can currently
be compiled to run on.
use Encode 'from_to';
my %ebcdic = ( 176 => 'cp37', 95 => 'cp1047', 106 => 'posix-bc' );
# $a is in EBCDIC code points
from_to($a, $ebcdic{ord '^'}, 'latin1');
# $a is ISO 8859-1 code points
and from Latin-1 code points to EBCDIC code points
use Encode 'from_to';
my %ebcdic = ( 176 => 'cp37', 95 => 'cp1047', 106 => 'posix-bc' );
# $a is ISO 8859-1 code points
from_to($a, 'latin1', $ebcdic{ord '^'});
# $a is in EBCDIC code points
For doing I/O it is suggested that you use the autotranslating features
of PerlIO, see L.
Since version 5.8 Perl uses the new PerlIO I/O library. This enables
you to use different encodings per IO channel. For example you may use
use Encode;
open($f, ">:encoding(ascii)", "test.ascii");
print $f "Hello World!\n";
open($f, ">:encoding(cp37)", "test.ebcdic");
print $f "Hello World!\n";
open($f, ">:encoding(latin1)", "test.latin1");
print $f "Hello World!\n";
open($f, ">:encoding(utf8)", "test.utf8");
print $f "Hello World!\n";
to get four files containing "Hello World!\n" in ASCII, CP 37 EBCDIC,
ISO 8859-1 (Latin-1) (in this example identical to ASCII since only ASCII
characters were printed), and
UTF-EBCDIC (in this example identical to normal EBCDIC since only characters
that don't differ between EBCDIC and UTF-EBCDIC were printed). See the
documentation of Encode::PerlIO for details.
As the PerlIO layer uses raw IO (bytes) internally, all this totally
ignores things like the type of your filesystem (ASCII or EBCDIC).
=head1 SINGLE OCTET TABLES
The following tables list the ASCII and Latin 1 ordered sets including
the subsets: C0 controls (0..31), ASCII graphics (32..7e), delete (7f),
C1 controls (80..9f), and Latin-1 (a.k.a. ISO 8859-1) (a0..ff). In the
table non-printing control character names as well as the Latin 1
extensions to ASCII have been labelled with character names roughly
corresponding to I albeit with
substitutions such as s/LATIN// and s/VULGAR// in all cases,
s/CAPITAL LETTER// in some cases, and s/SMALL LETTER ([A-Z])/\l$1/
in some other cases (the C pragma names unfortunately do
not list explicit names for the C0 or C1 control characters). The
"names" of the C1 control set (128..159 in ISO 8859-1) listed here are
somewhat arbitrary. The differences between the 0037 and 1047 sets are
flagged with ***. The differences between the 1047 and POSIX-BC sets
are flagged with ###. All ord() numbers listed are decimal. If you
would rather see this table listing octal values then run the table
(that is, the pod version of this document since this recipe may not
work with a pod2_other_format translation) through:
=over 4
=item recipe 0
=back
perl -ne 'if(/(.{33})(\d+)\s+(\d+)\s+(\d+)\s+(\d+)/)' \
-e '{printf("%s%-9o%-9o%-9o%o\n",$1,$2,$3,$4,$5)}' perlebcdic.pod
If you want to retain the UTF-x code points then in script form you
might want to write:
=over 4
=item recipe 1
=back
open(FH,") {
if (/(.{33})(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\.?(\d*)\s+(\d+)\.?(\d*)/) {
if ($7 ne '' && $9 ne '') {
printf("%s%-9o%-9o%-9o%-9o%-3o.%-5o%-3o.%o\n",$1,$2,$3,$4,$5,$6,$7,$8,$9);
}
elsif ($7 ne '') {
printf("%s%-9o%-9o%-9o%-9o%-3o.%-5o%o\n",$1,$2,$3,$4,$5,$6,$7,$8);
}
else {
printf("%s%-9o%-9o%-9o%-9o%-9o%o\n",$1,$2,$3,$4,$5,$6,$8);
}
}
}
If you would rather see this table listing hexadecimal values then
run the table through:
=over 4
=item recipe 2
=back
perl -ne 'if(/(.{33})(\d+)\s+(\d+)\s+(\d+)\s+(\d+)/)' \
-e '{printf("%s%-9X%-9X%-9X%X\n",$1,$2,$3,$4,$5)}' perlebcdic.pod
Or, in order to retain the UTF-x code points in hexadecimal:
=over 4
=item recipe 3
=back
open(FH,") {
if (/(.{33})(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\.?(\d*)\s+(\d+)\.?(\d*)/) {
if ($7 ne '' && $9 ne '') {
printf("%s%-9X%-9X%-9X%-9X%-2X.%-6X%-2X.%X\n",$1,$2,$3,$4,$5,$6,$7,$8,$9);
}
elsif ($7 ne '') {
printf("%s%-9X%-9X%-9X%-9X%-2X.%-6X%X\n",$1,$2,$3,$4,$5,$6,$7,$8);
}
else {
printf("%s%-9X%-9X%-9X%-9X%-9X%X\n",$1,$2,$3,$4,$5,$6,$8);
}
}
}
incomp- incomp-
8859-1 lete lete
chr 0819 0037 1047 POSIX-BC UTF-8 UTF-EBCDIC
------------------------------------------------------------------------------------
0 0 0 0 0 0
1 1 1 1 1 1
2 2 2 2 2 2
3 3 3 3 3 3
4 55 55 55 4 55
5 45 45 45 5 45
6 46 46 46 6 46
7 47 47 47 7 47
8 22 22 22 8 22
9 5 5 5 9 5
10 37 21 21 10 21 ***
11 11 11 11 11 11