Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to content

Commit 55613bf

Browse files
adunstanadunstan
committed
Close previously open holes for invalidly encoded data to enter the
database via builtin functions, as recently discussed on -hackers. chr() now returns a character in the database encoding. For UTF8 encoded databases the argument is treated as a Unicode code point. For other multi-byte encodings the argument must designate a strict ascii character, or an error is raised, as is also the case if the argument is 0. ascii() is adjusted so that it remains the inverse of chr(). The two argument form of convert() is gone, and the three argument form now takes a bytea first argument and returns a bytea. To cover this loss three new functions are introduced: . convert_from(bytea, name) returns text - converts the first argument from the named encoding to the database encoding . convert_to(text, name) returns bytea - converts the first argument from the database encoding to the named encoding . length(bytea, name) returns int - gives the length of the first argument in characters in the named encoding
1 parent 8544110 commit 55613bf

File tree

9 files changed

+318
-58
lines changed

9 files changed

+318
-58
lines changed

‎doc/src/sgml/func.sgml

Lines changed: 68 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
<!-- $PostgreSQL: pgsql/doc/src/sgml/func.sgml,v 1.395 2007/09/14 15:53:48 momjian Exp $ -->
1+
<!-- $PostgreSQL: pgsql/doc/src/sgml/func.sgml,v 1.396 2007/09/18 17:41:16 adunstan Exp $ -->
22

33
<chapter id="functions">
44
<title>Functions and Operators</title>
@@ -1122,13 +1122,14 @@
11221122
<row>
11231123
<entry><literal><function>convert</function>(<parameter>string</parameter>
11241124
using <parameter>conversion_name</parameter>)</literal></entry>
1125-
<entry><type>text</type></entry>
1125+
<entry><type>bytea</type></entry>
11261126
<entry>
11271127
Change encoding using specified conversion name. Conversions
11281128
can be defined by <command>CREATE CONVERSION</command>. Also
11291129
there are some pre-defined conversion names. See <xref
11301130
linkend="conversion-names"> for available conversion
1131-
names.
1131+
names. The <parameter>string</parameter> must be valid in the
1132+
source encoding.
11321133
</entry>
11331134
<entry><literal>convert('PostgreSQL' using iso_8859_1_to_utf8)</literal></entry>
11341135
<entry><literal>'PostgreSQL'</literal> in UTF8 (Unicode, 8-bit) encoding</entry>
@@ -1244,6 +1245,12 @@
12441245
<indexterm>
12451246
<primary>chr</primary>
12461247
</indexterm>
1248+
<indexterm>
1249+
<primary>convert_from</primary>
1250+
</indexterm>
1251+
<indexterm>
1252+
<primary>convert_to</primary>
1253+
</indexterm>
12471254
<indexterm>
12481255
<primary>decode</primary>
12491256
</indexterm>
@@ -1319,7 +1326,12 @@
13191326
<row>
13201327
<entry><literal><function>ascii</function>(<parameter>string</parameter>)</literal></entry>
13211328
<entry><type>int</type></entry>
1322-
<entry><acronym>ASCII</acronym> code of the first byte of the argument</entry>
1329+
<entry>
1330+
<acronym>ASCII</acronym> code of the first character of the argument.
1331+
For <acronym>UTF8</acronym> returns the Unicode code point of the character.
1332+
For other multi-byte encodings. the argument must be a strictly
1333+
<acronym>ASCII</acronym> character.
1334+
</entry>
13231335
<entry><literal>ascii('x')</literal></entry>
13241336
<entry><literal>120</literal></entry>
13251337
</row>
@@ -1340,29 +1352,61 @@
13401352
<row>
13411353
<entry><literal><function>chr</function>(<type>int</type>)</literal></entry>
13421354
<entry><type>text</type></entry>
1343-
<entry>Character with the given <acronym>ASCII</acronym> code</entry>
1355+
<entry>
1356+
Character with the given code. For <acronym>UTF8</acronym> the argument is
1357+
treated as a Unicode code point. For other multi-byte encodings the argument
1358+
must designate a strictly <acronym>ASCII</acronym> character.
1359+
</entry>
13441360
<entry><literal>chr(65)</literal></entry>
13451361
<entry><literal>A</literal></entry>
13461362
</row>
13471363

13481364
<row>
13491365
<entry>
1350-
<literal><function>convert</function>(<parameter>string</parameter> <type>text</type>,
1351-
<optional><parameter>src_encoding</parameter> <type>name</type>,</optional>
1366+
<literal><function>convert</function>(<parameter>string</parameter> <type>bytea</type>,
1367+
<parameter>src_encoding</parameter> <type>name</type>,
13521368
<parameter>dest_encoding</parameter> <type>name</type>)</literal>
13531369
</entry>
1354-
<entry><type>text</type></entry>
1370+
<entry><type>bytea</type></entry>
13551371
<entry>
13561372
Convert string to <parameter>dest_encoding</parameter>.
13571373
The original encoding is specified by
1358-
<parameter>src_encoding</parameter>. If
1359-
<parameter>src_encoding</parameter> is omitted, database
1360-
encoding is assumed.
1374+
<parameter>src_encoding</parameter>. The <parameter>string</parameter>
1375+
must be valid in this encoding.
13611376
</entry>
13621377
<entry><literal>convert( 'text_in_utf8', 'UTF8', 'LATIN1')</literal></entry>
13631378
<entry><literal>text_in_utf8</literal> represented in ISO 8859-1 encoding</entry>
13641379
</row>
13651380

1381+
<row>
1382+
<entry>
1383+
<literal><function>convert_from</function>(<parameter>string</parameter> <type>bytea</type>,
1384+
<parameter>src_encoding</parameter> <type>name</type>)</literal>
1385+
</entry>
1386+
<entry><type>text</type></entry>
1387+
<entry>
1388+
Convert string to the database encoding.
1389+
The original encoding is specified by
1390+
<parameter>src_encoding</parameter>. The <parameter>string</parameter>
1391+
must be valid in this encoding.
1392+
</entry>
1393+
<entry><literal>convert_from( 'text_in_utf8', 'UTF8')</literal></entry>
1394+
<entry><literal>text_in_utf8</literal> represented in the current database encoding</entry>
1395+
</row>
1396+
1397+
<row>
1398+
<entry>
1399+
<literal><function>convert_to</function>(<parameter>string</parameter> <type>text</type>,
1400+
<parameter>dest_encoding</parameter> <type>name</type>)</literal>
1401+
</entry>
1402+
<entry><type>text</type></entry>
1403+
<entry>
1404+
Convert string to <parameter>dest_encoding</parameter>.
1405+
</entry>
1406+
<entry><literal>convert_to( 'some text', 'UTF8')</literal></entry>
1407+
<entry><literal>some text</literal> represented in the UTF8 encoding</entry>
1408+
</row>
1409+
13661410
<row>
13671411
<entry>
13681412
<literal><function>decode</function>(<parameter>string</parameter> <type>text</type>,
@@ -1415,6 +1459,19 @@
14151459
<entry><literal>4</literal></entry>
14161460
</row>
14171461

1462+
<row>
1463+
<entry><literal><function>length</function>(<parameter>string</parameter><type>bytea</type>,
1464+
<parameter>encoding</parameter> <type>name</type> )</literal></entry>
1465+
<entry><type>int</type></entry>
1466+
<entry>
1467+
Number of characters in <parameter>string</parameter> in the
1468+
given <parameter>encoding</parameter>. The
1469+
<parameter>string</parameter> must be valid in this encoding.
1470+
</entry>
1471+
<entry><literal>length('jose', 'UTF8')</literal></entry>
1472+
<entry><literal>4</literal></entry>
1473+
</row>
1474+
14181475
<row>
14191476
<entry>
14201477
<literal><function>lpad</function>(<parameter>string</parameter> <type>text</type>,

‎src/backend/catalog/pg_conversion.c

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
*
99
*
1010
* IDENTIFICATION
11-
* $PostgreSQL: pgsql/src/backend/catalog/pg_conversion.c,v 1.36 2007/02/27 23:48:07 tgl Exp $
11+
* $PostgreSQL: pgsql/src/backend/catalog/pg_conversion.c,v 1.37 2007/09/18 17:41:17 adunstan Exp $
1212
*
1313
*-------------------------------------------------------------------------
1414
*/
@@ -282,7 +282,10 @@ FindConversion(const char *conname, Oid connamespace)
282282
* CONVERT <left paren> <character value expression>
283283
* USING <form-of-use conversion name> <right paren>
284284
*
285-
* TEXT convert_using(TEXT string, TEXT conversion_name)
285+
* BYTEA convert_using(TEXT string, TEXT conversion_name)
286+
*
287+
* bytea is returned so we don't give a value that is
288+
* not valid in the database encoding.
286289
*/
287290
Datum
288291
pg_convert_using(PG_FUNCTION_ARGS)
@@ -344,5 +347,5 @@ pg_convert_using(PG_FUNCTION_ARGS)
344347
pfree(result);
345348
pfree(str);
346349

347-
PG_RETURN_TEXT_P(retval);
350+
PG_RETURN_BYTEA_P(retval);
348351
}

‎src/backend/utils/adt/oracle_compat.c

Lines changed: 137 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@
99
*
1010
*
1111
* IDENTIFICATION
12-
* $PostgreSQL: pgsql/src/backend/utils/adt/oracle_compat.c,v 1.70 2007/02/27 23:48:08 tgl Exp $
12+
* $PostgreSQL: pgsql/src/backend/utils/adt/oracle_compat.c,v 1.71 2007/09/18 17:41:17 adunstan Exp $
1313
*
1414
*-------------------------------------------------------------------------
1515
*/
@@ -1246,18 +1246,71 @@ translate(PG_FUNCTION_ARGS)
12461246
*
12471247
* Returns the decimal representation of the first character from
12481248
* string.
1249+
* If the string is empty we return 0.
1250+
* If the database encoding is UTF8, we return the Unicode codepoint.
1251+
* If the database encoding is any other multi-byte encoding, we
1252+
* return the value of the first byte if it is an ASCII character
1253+
* (range 1 .. 127), or raise an error.
1254+
* For all other encodings we return the value of the first byte,
1255+
* (range 1..255).
12491256
*
12501257
********************************************************************/
12511258

12521259
Datum
12531260
ascii(PG_FUNCTION_ARGS)
12541261
{
12551262
text *string = PG_GETARG_TEXT_P(0);
1263+
int encoding = GetDatabaseEncoding();
1264+
unsigned char *data;
12561265

12571266
if (VARSIZE(string) <= VARHDRSZ)
12581267
PG_RETURN_INT32(0);
12591268

1260-
PG_RETURN_INT32((int32) *((unsigned char *) VARDATA(string)));
1269+
data = (unsigned char *) VARDATA(string);
1270+
1271+
if (encoding == PG_UTF8 && *data > 127)
1272+
{
1273+
/* return the code point for Unicode */
1274+
1275+
int result = 0, tbytes = 0, i;
1276+
1277+
if (*data >= 0xF0)
1278+
{
1279+
result = *data & 0x07;
1280+
tbytes = 3;
1281+
}
1282+
else if (*data >= 0xE0)
1283+
{
1284+
result = *data & 0x0F;
1285+
tbytes = 2;
1286+
}
1287+
else
1288+
{
1289+
Assert (*data > 0xC0);
1290+
result = *data & 0x1f;
1291+
tbytes = 1;
1292+
}
1293+
1294+
Assert (tbytes > 0);
1295+
1296+
for (i = 1; i <= tbytes; i++)
1297+
{
1298+
Assert ((data[i] & 0xC0) == 0x80);
1299+
result = (result << 6) + (data[i] & 0x3f);
1300+
}
1301+
1302+
PG_RETURN_INT32(result);
1303+
}
1304+
else
1305+
{
1306+
if (pg_encoding_max_length(encoding) > 1 && *data > 127)
1307+
ereport(ERROR,
1308+
(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
1309+
errmsg("requested character too large")));
1310+
1311+
1312+
PG_RETURN_INT32((int32) *data);
1313+
}
12611314
}
12621315

12631316
/********************************************************************
@@ -1270,19 +1323,96 @@ ascii(PG_FUNCTION_ARGS)
12701323
*
12711324
* Purpose:
12721325
*
1273-
* Returns the character having the binary equivalent to val
1326+
* Returns the character having the binary equivalent to val.
1327+
*
1328+
* For UTF8 we treat the argumwent as a Unicode code point.
1329+
* For other multi-byte encodings we raise an error for arguments
1330+
* outside the strict ASCII range (1..127).
1331+
*
1332+
* It's important that we don't ever return a value that is not valid
1333+
* in the database encoding, so that this doesn't become a way for
1334+
* invalid data to enter the database.
12741335
*
12751336
********************************************************************/
12761337

12771338
Datum
12781339
chr(PG_FUNCTION_ARGS)
12791340
{
1280-
int32 cvalue = PG_GETARG_INT32(0);
1341+
uint32 cvalue = PG_GETARG_UINT32(0);
12811342
text *result;
1343+
int encoding = GetDatabaseEncoding();
1344+
1345+
if (encoding == PG_UTF8 && cvalue > 127)
1346+
{
1347+
/* for Unicode we treat the argument as a code point */
1348+
int bytes ;
1349+
char *wch;
12821350

1283-
result = (text *) palloc(VARHDRSZ + 1);
1284-
SET_VARSIZE(result, VARHDRSZ + 1);
1285-
*VARDATA(result) = (char) cvalue;
1351+
/* We only allow valid Unicode code points */
1352+
if (cvalue > 0x001fffff)
1353+
ereport(ERROR,
1354+
(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
1355+
errmsg("requested character too large for encoding: %d",
1356+
cvalue)));
1357+
1358+
if (cvalue > 0xffff)
1359+
bytes = 4;
1360+
else if (cvalue > 0x07ff)
1361+
bytes = 3;
1362+
else
1363+
bytes = 2;
1364+
1365+
result = (text *) palloc(VARHDRSZ + bytes);
1366+
SET_VARSIZE(result, VARHDRSZ + bytes);
1367+
wch = VARDATA(result);
1368+
1369+
if (bytes == 2)
1370+
{
1371+
wch[0] = 0xC0 | ((cvalue >> 6) & 0x1F);
1372+
wch[1] = 0x80 | (cvalue & 0x3F);;
1373+
}
1374+
else if (bytes == 3)
1375+
{
1376+
wch[0] = 0xE0 | ((cvalue >> 12) & 0x0F);
1377+
wch[1] = 0x80 | ((cvalue >> 6) & 0x3F);
1378+
wch[2] = 0x80 | (cvalue & 0x3F);
1379+
}
1380+
else
1381+
{
1382+
wch[0] = 0xF0 | ((cvalue >> 18) & 0x07);
1383+
wch[1] = 0x80 | ((cvalue >> 12) & 0x3F);
1384+
wch[2] = 0x80 | ((cvalue >> 6) & 0x3F);
1385+
wch[3] = 0x80 | (cvalue & 0x3F);
1386+
}
1387+
1388+
}
1389+
1390+
else
1391+
{
1392+
bool is_mb;
1393+
1394+
/* Error out on arguments that make no sense or that we
1395+
* can't validly represent in the encoding.
1396+
*/
1397+
1398+
if (cvalue == 0)
1399+
ereport(ERROR,
1400+
(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
1401+
errmsg("null character not permitted")));
1402+
1403+
is_mb = pg_encoding_max_length(encoding) > 1;
1404+
1405+
if ((is_mb && (cvalue > 255)) || (! is_mb && (cvalue > 127)))
1406+
ereport(ERROR,
1407+
(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
1408+
errmsg("requested character too large for encoding: %d",
1409+
cvalue)));
1410+
1411+
1412+
result = (text *) palloc(VARHDRSZ + 1);
1413+
SET_VARSIZE(result, VARHDRSZ + 1);
1414+
*VARDATA(result) = (char) cvalue;
1415+
}
12861416

12871417
PG_RETURN_TEXT_P(result);
12881418
}

0 commit comments

Comments
 (0)