Simple Comments Release Notes: v.910 (2/2)

[previous]

Simple Comments: v.910

Caching Configurations

In a standard Perl environment, Simple Comments will read in its configuration file settings every time the script runs; i.e., if the user reloads a page that displays comments, then the configuration file is reloaded with each call to the comments.pl script. While not very efficient (after all, the configuration file will rarely change after the initial setup), a standard Perl environment offers us no choice; as the interpreter and script are freshly reloaded each time the script is called, and therefore the configuration parameters must be freshly read on each invocation.

The situation is different, however, in a mod_perl environment. In mod_perl, the interpreter and script are not freshly loaded each time the page is called; instead, they are cached, along with the current state of all of their defined variables. Because of this, we can make our Simple Comments script more efficient specifically for mod_perl by checking the state of our configuration variable entry before loading it; i.e., something like this:

our $config ||= get_configuration($default_config_file);

Now, if $config is already loaded we skip its assignment, and if it's not we initialize it; mod_perl will thus load the configuration settings the first time the script runs, but will reuse those configuration settings--avoiding the need to reread the configuration file and reload the $config hash every time the script is called. These parameters will remain accessible in memory theoretically until the server is rebooted, though in reality it will more likely be until the server reaches the maximum number of child requests and kills/restarts each child process. In a standard Perl environment, the $config variable will still be initialized with each launch of the script; since it's unable to "stick around" between script calls anyways.

If we stopped here, however, we would neglect a potentially serious problem in this new design. Specifically, what if you change the configuration file? The script will potentially not even recognize your changes until you restart the server--something you more than likely do not want to do on a routine basis. To force the scripts to reload the configuration file as necessary, you can check the date modified time of the configuration file itself. If it's changed since the last time you needed it, then you reload the parameters. Otherwise, just keep and use the parameters you already have. This is the method we've chosen to use in Simple Comments; and the entire process is more adequately and thoroughly described in this excerpt from Practical mod_perl:

Loading and Reloading Modules

utf8 and the Unexpected SWASHNEW Error

Having tested the script on my development box (which houses Perl 5.8.7), I began testing it on a 5.6 Perl server. When trying to run the administration script with the new built-in user authentication in Perl 5.6.1, I received the following error:

Can't locate object method "SWASHNEW" via package "utf8" 
(perhaps you forgot to load "utf8"?)

I knew that version 5.6.1 required the use of the utf8 pragma to ensure that certain character-type operations were performed in a utf8-aware manner. And I also knew that version 5.8 did away, for the most part, with the need to use utf8;, reserving it only for situations where the Perl code itself utilized utf8 characters:

"Starting from Perl 5.8.0, the use of use utf8 is no longer necessary. In earlier releases the utf8 pragma was used to declare that operations in the current block or file would be Unicode-aware. This model was found to be wrong, or at least clumsy: the "Unicodeness" is now carried with the data, instead of being attached to the operations. Only one case remains where an explicit use utf8 is needed: if your Perl script itself is encoded in UTF-8, you can use UTF-8 in your identifier names, and in string and regular expression literals, by saying use utf8 . This is not the default because scripts with legacy 8-bit data in them would break."
https://perldoc.perl.org/perluniintro.html

Finally, I was not--or so I thought--utilizing data that needed to use utf8-aware processing. As a result, the above error had me stumped.

As we've all often done, I began setting up test cases and slashing down the code in an attempt to isolate the root cause of the error. The following reduction--which produces the above mentioned SWASHNEW error in a Perl 5.6.1 environment--is the result of that effort:

use strict;
my $test = pack "U0A*", "Dan";
print uc($test), "\n";
sub do_nothing {
  utf8::encode('foobar');
}

The explanation of the problem is as follows:

• Beginning with Perl 5.6, a scalar variable can be marked internally as a utf8 encoded variable. Perl flips this flag automatically when it notes that the variable contains utf8 specific characters. One way to force a value to be considered utf8 is to pack it with the "U0" prefix.

  In the actual Simple Comments code, I wasn't forcing any values to be considered utf8, but what hadn't occurred to me was the fact that XML::Simple was automatically returning the values that it read from my XML files already marked as being utf8! In retrospect this makes sense; as our configuration files (and our comment files) didn't specify a specific encoding and XML::Parser was correct in defaulting to returning utf8 data.

  An excellent source of further information on the use of XML specifically in Perl can be found in the Perl-XML FAQ.

• When a particular string--such as $test--is marked as utf8, certain Perl functions, i.e., those that must know and correctly deal with the character encodings of the data contained within the parameters, will behave slightly differently when the utf8 value is passed to it. A classic example is length; which for a utf8 string returns the number of characters that are included in a particular string, and not the number of bytes (which, in a utf8 string, may be different than the character count, since some characters may be represented by more than one byte).

  In the new version of Simple Comments, we convert some of our configuration entries--i.e., those returned by XML::Simple, to uppercase, to facilitate case-insensitive matching. To do this, we use Perl's built-in uc function, which is utf8-aware.

• As we mentioned earlier, in Perl 5.8 the specific need to use utf8 was removed; you only need to use it if your script itself includes utf8 encodings. Routines that are accessed through utf8 can be referenced directly; i.e., in Perl 5.8 it's legal to say utf8::encode('foobar'), without having previously used utf8; and in fact, this is the recommended approach. In Simple Comments, we weren't directly referencing any of the utf8 subroutines; however, some digging encountered an included module that was.

• Finally, the clincher. If all of the above parameters are true, then a bug in Perl 5.6 will trigger the utf8 aware functions to perform operations that are located in the utf8 module--including SWASHNEW--even if you didn't use the utf8 module. Have a look at the reduction above again:
  1. We have a variable that we know is marked utf8 ($test).
  2. We pass that variable to a utf8 aware function (uc).
  3. Elsewhere in the code, we reference a utf8 function without having actually used utf8. It doesn't matter if the utf8 function is actually executed in the code; the reference just needs to exist somewhere in the code.

This, in fact, is the sequence of events that lead to the SWASHNEW error we noted above in Simple Comments. We retrieved utf8-marked values (via XML::Simple), and attempted to perform string operations on them (uc). Meanwhile, the URI::Escape module, which no doubt had at some point been updated separately from the Perl version on our particular test server, just happened to have a bare utf8::encode() call in it, satisfying all the necessary conditions for the appearance of the error.

Having identified the problem, how were we to fix it? Given the general recommendation that we should not use utf8; in Perl 5.8 scripts unless we actually have utf8 characters in our script code, we resisted just adding a blanket use utf8; to the scripts. Instead, we opted to conditionally add the utf8 pragma, based on the Perl version in use:

if ($[ < 5.008) { use utf8; }

Another possibility that might work for you is to perform a typical untaint of the variables by replacing their values with $1, $2 etc. references; i.e., something like:

if ($test =~ /^([A-Za_z]*)$/) {
   $test = $1;
} 
else {
   die "ERROR IN INPUT!\n";
}

In Simple Comments, we now incorporate both fixes: We perform our untaint checking operation on the values before we attempt to use functions such as uc, and we conditionally include the utf8 pragma, just to be safe.

Conclusion

It's my hope that you will continue to find the Simple Comments script itself useful, and/or the developmental notes to helpful for your own Perl projects. Please feel free to contact me if you have any clarifications, suggestions or reqests for improvements!

[previous]

Created: September 13, 2006
Revised: September 13, 2006

URL: https://webreference.com/programming/perl/comments/v.910/2.html