Philadelphia Reflections: Website Development

Website Development

The website technology supporting Philadelphia Reflections is PHP, MySQL and DHTML. The web hosting service is Internet Planners.

The primary purpose of this website is to deliver high quality content on the subjects of Philadelphia, Philadelphia History, medicine, medical economics and other subjects of interest to its author, Dr. George R. Fisher.

However, early in 2006 the site was attacked by spammers who broke in using security holes in the previous implementation of PHP. In the subsequent reconstruction of the site, there's been an opportunity to try out lots of new technology and techniques, some of which are detailed here.

Parsing name-value pair attributes in an HTML tag

Not only do the attributes in an HTML tag come in random order but many are optional

Here's a regex solution:

<?php
function tagAttr($matches) {print_r($matches);}

$string = '<img src="/images/picture.jpg" width="300" class="left" alt="alt keywords" />';

$foo	= preg_replace_callback(
'/<img\b(?>\s+(?:alt="([^"]*)"|class="([^"]*)"|style="([^"]*)"|src="([^"]*)"|height="([^"]*)"|width="([^"]*)")|[^\s>]+|\s+)*>/i',
"tagAttr",
$string);
?>

Produces the following:

Array
(
    [0] => <img src="/images/picture.jpg" width="300" class="left" alt="alt keywords" />
    [1] => alt keywords
    [2] => left
    [3] => 
    [4] => /images/picture.jpg
    [5] => 
    [6] => 300
)

The regex is a series of alternating sequences; so, add href="([^"]*)"| in front of alt="([^"]*)" to select an additional attribute.

My thanks (a) to Flagrant Badassery for putting me onto the idea and (b) to http://centricle.com/tools/html-entities/ for HTML encoding

Regex URL Matching

On this site we check for the existence of a URL whenever an entry is updated

There are two key technologies at work

A PHP function that checks whether a URL is valid (thanks to marufit at gmail dot com in the PHP Manual)
Regex (regular expression) in a preg_replace_callback routine; this one is mine, all mine


function url_exists($url) 
{
// 
// checks whether a URL actually exists on the Internet
//
$handle   = curl_init($url);
if (false === $handle)
   {
    return false;
   }
curl_setopt($handle, CURLOPT_HEADER, false);
curl_setopt($handle, CURLOPT_FAILONERROR, true); 
curl_setopt($handle, CURLOPT_NOBODY, true);
curl_setopt($handle, CURLOPT_RETURNTRANSFER, false);
$connectable = curl_exec($handle);
curl_close($handle);   
return $connectable;
}


function aExists($matches)
{
//
// function called by preg_replace_callback
//
// $matches[0] is the complete match
// $matches[1] the match for the first subpattern
//	enclosed in '(...)' and so on

//
// checks to see if a regular link exists
// something similar is done for img src= also
//

$srcURL = $matches[3];
		
if (url_exists($srcURL)) {do something; return "";}  
else {do something else; return "";}
}

$foo = preg_replace_callback(
            '/(.*?)(<a .*?href=")([^"]*)("[^>]*>)(.*?)(<\/a>)/i',
            "aExists",
            $source_string);

(my thanks to http://centricle.com/tools/html-entities/ for HTML encoding)

HTML Forms

How do you (a) open a form when a radio button is clicked (b) in a new window?

Here's how it's done on this website.

The radio button is activated by a little JavaScript routine
The new window is simply a matter of including the target attribute in the form tag

<html>
<head>

<script type="text/javascript">

	/* javascript function called by the radio buttons 
	     to submit the form when clicked */

	function formSubmit()
	{
	document.getElementById("form_x").submit()
	}
	
</script>

</head>
<body>

  <form name="form_x" id="form_x"
	action="some_routine.php"
	target="newIMGwin"
	method="post" 
	style="whatever">

	<fieldset>
	<legend>legend surrounding the form</legend>

	<input type="radio" name="key" value="1269" onclick="formSubmit()" />

	</fieldset>
  </form>

</body>
</html>

(my thanks to http://centricle.com/tools/html-entities/ for HTML encoding)

XHTML vs. HTML

The markup language used by web browsers continues to evolve. The most current version (as of August 2006) is XHTML 1.1, an XML version of HTML.

Many browsers, most particularly IE, do not support XHTML. Technically speaking, they support only the "text/html" mime type, not "application/xhtml+xml". Lots of web developers have gone to the trouble of sticking closing tags ( />) in their BR, HR, META and INPUT tags and a DOCTYPE at the top but then serve the code as "text/html".

This produces a syntactic mish mash which is almost certainly worse than using strict HTML 4.01.

Why "worse"? Because of the possibility of unintended results from providing incorrect instructions to the browser. If you care about the output produced by the browser, which most developers and content providers emphatically do, then you have to be careful about what instructions you give the browser. You simply cannot count on getting what you want if what you're telling the browser to do is syntactically incorrect.

However, it's a little difficult to see just what good XHTML is:

There are rumors that it renders the non-image portion of a page as much as 50% faster than HTML, but what with gzip and broadband being pretty common these days, it's hard to see that as an especially compelling reason to be bothered.
Furthermore, those browsers that do render XHTML (Mozilla, Firefox) are very picky about syntax and blow up much too easily.
And the claim that XHTML is the way to get your web pages onto cell phones and toaster ovens leaves me cold. It's just not believable that the format required for these special devices will be the same as for a computer monitor.

Internet cognoscenti speak disparagingly of "tag soup" but the Internet is a lot more about content than it is about syntax, so who really cares?

Well, somehow, I do. A little. Since we use PHP on this site, we have the opportunity to figure out what features are supported by a browser and render the correct types of tags, mime-types, etc.

Check out the headers and the page source in Mozilla to see it in action:

It renders XHTML 1.1 correctly whenever it encounters a browser that can support it
It uses output buffering (which demonstrably if illogically improves rendering response time)
It sends the whole thing using gzip compression if the browser will support it

<?php

#
# Both http://www.workingwith.me.uk/articles/scripting/mimetypes
# and http://keystonewebsites.com/articles/mime_type.php
#
# ... show how to serve the correct mime type and HTML prologue
#
# ... I prefer http://www.workingwith.me.uk/articles/scripting/mimetypes
#     because it serves XHTML 1.1 instead of XHTML 1.0 Transitional
#     and HTML 4.01 Loose
#
# http://www.hixie.ch/advocacy/xhtml Discusses using the wrong mime type.
#
# http://www.goer.org/ is a very interesting site on this subject
#
#
# I also include the privacy header created for me by http://www.p3pwiz.com/
# and I modified the fix_code function to include gzip.
#
# It is possible that we can get the server to do all our compression for us automatically.
# At this point I have not tested this but here are two references:
#
# http://elliottback.com/wp/archives/2006/01/12/http-gzip-compression-in-php/
# http://lists.evolt.org/archive/Week-of-Mon-20050228/170256.html
#

$charset = "iso-8859-1";
$mime = "text/html";

function fix_code($buffer)
	{
	#
	# I modified this for gzip
	#
	if (substr_count($_SERVER['HTTP_ACCEPT_ENCODING'], 'gzip'))
		{
		header("Content-Encoding: gzip"); // required to un-gzip to output
		return (gzencode(str_replace(" />", ">", $buffer),6,FORCE_GZIP));
		}
		else
			{
			return (str_replace(" />", ">", $buffer));
			}
	}

if(stristr($_SERVER["HTTP_ACCEPT"],"application/xhtml+xml")) {
   # if there's a Q value for "application/xhtml+xml" then also 
   # retrieve the Q value for "text/html"
   if(preg_match("/application\/xhtml\+xml;q=0(\.[1-9]+)/i",
                 $_SERVER["HTTP_ACCEPT"], $matches)) {
      $xhtml_q = $matches[1];
      if(preg_match("/text\/html;q=0(\.[1-9]+)/i",
                    $_SERVER["HTTP_ACCEPT"], $matches)) {
         $html_q = $matches[1];
         # if the Q value for XHTML is greater than or equal to that 
         # for HTML then use the "application/xhtml+xml" mimetype
         if($xhtml_q >= $html_q) {
            $mime = "application/xhtml+xml";
         }
      }
   # if there was no Q value, then just use the 
   # "application/xhtml+xml" mimetype
   } else {
      $mime = "application/xhtml+xml";
   }
}

# special check for the W3C_Validator
if (stristr($_SERVER["HTTP_USER_AGENT"],"W3C_Validator")) {
   $mime = "application/xhtml+xml";
}

# set the prolog_type according to the mime type which was determined
if($mime == "application/xhtml+xml")
	{
	#
	# I added this, G4
	#
	ob_start("ob_gzhandler");
	#
	#
	$prolog_type = "<?xml version='1.0' encoding='$charset' ?>
<!DOCTYPE html PUBLIC '-//W3C//DTD XHTML 1.1//EN' 'http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd'>
<html xmlns='http://www.w3.org/1999/xhtml' xml:lang='en'>\n\n";
	}
	else
		{
		ob_start("fix_code");
		$prolog_type = "<!DOCTYPE HTML PUBLIC '-//W3C//DTD HTML 4.01//EN' 
		'http://www.w3.org/TR/html4/strict.dtd'>
		<html lang='en'>\n\n";
		}

# finally, output the mime type and prolog type
header("Content-Type: $mime;charset=$charset");
header("Vary: Accept");

// privacy header created at http://www.p3pwiz.com/
header("P3P: policyref=\"http://www.philadelphia-reflections.com/w3c/p3p.xml\",
 CP=\"NID DSP NOI COR\"");

print $prolog_type;
?>

Here's an interesting article on Doctype Switching: http://gutfeldt.ch/matthias/articles/doctypeswitch.html

The Philadelphia Reflections webmaster: George IV

(my thanks to http://centricle.com/tools/html-entities/ for HTML encoding)

2007 PS:
It turns out that Firefox has a number of intolerable quirks in the way it displays pages presented to it using XHTML 1.1 and the application/xhtml+xml mime type. I was unable to figure out a satisfactory way of circumventing these bugs and so I have reverted to XHTML 1.0 Strict and the text/html mime type, which solves all the problems but annoys me quite a lot.

Geo Positioning

Geo Tagging refers to adding latitude and longitude information to websites and photographs. This has been around for a long time but it has taken the advent of Google Earth for it to really start to catch on.

This blog entry has geo meta tags that you can see if you look at the HTML source ("View > [Page] Source"). The input was as follows:

Address: 82 Devonshire St Boston MA

Lat: 42.3578 Lon: -71.0577

Descriptive Place Name: Fidelity Investments headquarters

Region: US-MA Country Code: US Country Name: United States

This creates meta tags in the HTML Header as follows:

<!-- geo tags for 82 Devonshire St Boston MA -->
<meta name="ICBM"          content="42.3578, -71.0577" />

<meta name="geo.country"   content="US" />
<meta name="geo.region"    content="US-MA" />
<meta name="geo.placename" content="Fidelity Investments headquarters" />
<meta name="geo.position"  content="42.3578; -71.0577" />

<meta name="tgn.name"      content="Fidelity Investments headquarters" />
<meta name="tgn.nation"    content="United States" />

To be precise you can go to the location with a GPS device and record the exact coordinates.
(see below for a discussion of GPX, the GPS transfer protocol)
Google Earth is a very good way to find coordinates of any spot on earth
and both GE itself and the KML Editor will allow you to pick up coordinates from GE.
You can find the lat and lon of a US address using GeoURL Header Generator
My Geo Position can also be helpful
But for most lookups, the easiest method is to use the Firefox Sidebar AddOn called Minimap; it's right in your browser so you don't have to switch back and forth. (The AddOn download is found here: Mini Map Sidebar).

The Region, Country Code and Country Name can be found here: ISO-3166-1 Country Names

geo.placename and tgn.name are often rendered as the city name but are intended to describe the geographical feature ("Pyramids of Giza" or something). This tag is optional.

HTML geo meta tags can be validated here:

There is a search engine of long standing that reads HTML geo meta tags and indexes the website based upon its location; for searching, it groups sites based on their geographic proximity: GeoURL.

Photographs can also contain geo meta data, so-called EXIF data (Firefox has an EXIF viewer AddOn).

JPEG is the most common image format and the easiest to deal with. The combination of Picassa2 and Google Earth allow you easily to add this information to your own photos.

The process of adding lat and lon to your photographs is this:

1. Select one or more photos in Picassa
2. Select Tools > GeoTag > GeoTag with Google Earth ...
3. This starts Google Earth and you can "fly" to the location of the picture

4. A small Picasa window will appear in Earth's lower-right corner displaying thumbnails of the pictures you selected; press the "Geotag" button.
5. When all of your pictures are tagged, press the "Done" button

Slowly, camera manufacturers are providing GPS capability. Some few have GPS devices built in and some others allow an external GPS device to be attached, although both Canon and Nikon are way behind the curve ... if you own either, you can essentially forget it: the best - lousy - solution is to carry around a GPS with you and synchronize the times ... ugly.

The Google Maps API allows maps to be embedded in a website as is done here. Google Maps API

The JavaScript required to embed the map on this page can also be seen in the HTML source ("View > [Page] Source"). In addition to JavaScript, you need a DIV with an ID of "map" or whatever is specified in the JavaScript document.getElementById entry, which specifies the height and width of the map to be displayed.

To embed these maps you must register with Google

In addition, there are extensions to ATOM and RSS to include lat and lon in your syndication feeds; there are three standards that I have found: GeoRSS (ATOM example) , W3C Geo (RSS example) and an "ICBM RSS Module". This website extends the namespaces of both its ATOM feed and its RSS feed to include all the tags.

Google, Yahoo and Microsoft all now support GeoRSS as a feed to their map programs. My sense of it is that KML is a richer protocol, allowing more features, but fundamentally all these XML variants do mostly the same thing.

Google Sitemaps can include links to KML files (and ATOM, now, too). Part of the sitemap generation on this site is some code that picks up every *.kml and *.kmz file in the /kml/ folder and adds them to our sitemap.xml file.

Google Earth is filled with delights, not the least of which is a Flight Simulator! Google Earth Flight Simulator Keyboard Controls

KML ( Keyhole Markup Language, Keyhole being the predecessor to Google Earth) is an XML protocol that allows you to incorporate Google Earth into graphical presentations. Google KML Overview

Google Earth Outreach helps you get started: Google Earth Outreach

An extraordinary collection of KML files you can view is found here: Spectacular satellite images of the world

I found a KML editor here: NorthGates' KML Editor for Windows. It's rudimentary but very handy for what it does do.

Here's the Google Earth tools list where I found the KML editor: EarthPlot Software Tools For Google Earth

The way we serve the KML in the link that connects to Google Earth from individual blogs uses the following PHP script as its base:

<?php

// See Google Earth's KML 2.1 Reference
// http://code.google.com/apis/kml/documentation/kml_tags_21.html

$lat			= $_GET['lat'];
$lon			= $_GET['lon'];
$placename		= $_GET['placename'];
	
$altitude		= $_GET['altitude'];
$range			= $_GET['range'];
$heading		= $_GET['heading'];
$tilt			= $_GET['tilt'];
	
if ($altitude	== NULL) {$altitude	= 0;}
if ($range	== NULL) {$range	= 1000;}
if ($heading	== NULL) {$heading	= 0;}
if ($tilt	== NULL) {$tilt		= 0;}
	
$description	= "<h3><font color=\"#ea9f20\"><a href=\"http://www.philadelphia-reflections.com/\">
		PHILADELPHIA REFLECTIONS</font></a></h3>
		<p>The musings of a Philadelphia physician who has served the community for six decades.</p>";
	
	
header('Content-Type: application/vnd.google-earth.kml+xml');
header('Content-Disposition: inline; filename="philadelphia-reflections.kml"');

echo '<?xml version="1.0" encoding="UTF-8"?>'; 

?>

<kml xmlns="http://earth.google.com/kml/2.0">

  <Placemark>
    <name><?php echo $placename; ?></name>
    <description>
        <![CDATA[<?php echo $description; ?>]]>
    </description>
    
    <LookAt>
      <longitude><?php echo $lon; ?></longitude>
      <latitude><?php echo $lat; ?></latitude>
      
      <altitude><?php echo $altitude; ?></altitude>
      <range><?php echo $range; ?></range>
      <tilt><?php echo $tilt; ?></tilt>
      <heading><?php echo $heading; ?></heading>
      
      <altitudeMode>relativeToGround</altitudeMode>
    </LookAt>
    
    <Point>
      <coordinates><?php echo "$lon,$lat,$altitude"; ?></coordinates>
    </Point>
    
  </Placemark>

</kml>

Of course, GPS devices are an integral part of this process of Geo Positioning. GPS devices are supposed to support the open-source protocol GPX,
which is an XML-based description of waypoints and routes. Wikipedia describes GPX here: GPS eXchange Format

The GPX protocol's official website is here: GPX The GPS Exchange Format

Google Earth supports raw GPX (File > Open ...) and when you open a GPX file in Google Earth, it converts it to KML. But if you want stand alone programs to do this:

Convert from GPX to KML: GPS to Google Earth KMZ / KML converter
Convert from KML to GPX: Google Earth KMZ / KML to GPS converter

If you have non-standard GPS data, you may want to have a look at GPS Babel for conversion of native GPS formats as well as the GPS Utility and G7ToWin

A nice blog on these things relative to Google Maps is here: Using XSL to Transform Google Earth (KML) and GPX to Google Maps API

At Philadelphia Reflections, we are creating tours by carrying a GPS and a camera around on our travels. The GPS track becomes a path and waypoints become placemarks. When you come home, download the GPS data in GPX format and open up the GPX file in Google Earth. Use Google Earth to edit the placemark balloons, including pictures and text.

There are many, many sightseeing blogs around that take you to interesting places on Google Maps and Google Earth. A place to start looking is Sightseeing with Google Satellite Maps

Somehow, the concept of "mashup" is related to all of this but it sort of sounds like the term "multimedia" a few years ago ... fancy in concept but somewhat vague in reality.

Google has a Mashup Editor and Wikipedia has a definition but it's not clear what it all adds up to.

(my thanks to http://centricle.com/tools/html-entities/ for HTML encoding)

Send a KML file from disk using PHP

Sending a kml or kmz disk file is as easy as clicking on it. But different browsers react differently, some asking you which program to use others storing the file on your desk top, etc. Preprocessing the file through PHP can reduce some of these annoyances.

<?php

//
// reads and sends a kml or kmz file
// located in /whatever/kml/
//
// calling protocol:
// this-program.php?file=somefile.kml
//

// read the input and check that it's a kmz or kml file
// ....................................................

$kml_file	= $_GET['file'];

if (($kml_file === NULL) or ($kml_file == "")) {exit ("error message");}
		
if ((substr($kml_file, -4) != ".kmz") AND (substr($kml_file, -4) != ".kml"))	
	{
	exit ("error message");
	}


// prepend the file path information to the file name and check that it exists
// ...........................................................................

$kml_file_name = "/whatever/kml/" . $kml_file;
	
if (!file_exists($kml_file_name)) { die ("error message");}


// send out the HTTP header information followed by the file contents
// ..................................................................
	
header("Cache-Control: no-cache, no-store, must-revalidate"); // trying to keep from getting the files stored on the local computer
header("Expires: Mon, 26 Jul 1997 05:00:00 GMT");

if (substr($kml_file, -4) == ".kml") {header('Content-Type: application/vnd.google-earth.kml+xml');}
if (substr($kml_file, -4) == ".kmz") {header('Content-Type: application/vnd.google-earth.kmz');}

header("Content-Disposition: inline");
header("Content-Description: KML or KMZ data intended for Google Earth");

readfile ($kml_file_name);

?>

Process .htm and .html as php

It is sometimes helpful to include php scripting in files that do not have the file extension of php.

There is quite a lot of discussion on the web about this but at least on this server the answer is not what most people think.

In the .htaccess file in the root folder include these two lines:

AddType application/x-httpd-php .html .php .htm
AddHandler application/x-httpd-php .html .php

Static vs Dynamic URLs

It used to be that no spiders or search engines could index a dynamic URL, namely one that contained a "?" followed by parameters to be used by PHP, ASP or other server-side scripting languages to drive a website using a database.

Nowadays, Google and Yahoo seem to do a perfectly fine job of indexing dynamic URLs but Google has a disclaimer warning that it may still encounter problems with dynamic URLs and the SEO literature is still full of warnings that other spiders and search engines may be blind to everything to the right of the "?".

Furthermore, a *.php extension is an invitation to bad guys to try to break in and wreak many sorts of havoc: this site was hacked by Nigerians a few years ago using PHP tricks and they managed to use it as an email factory until our ISP shut us down. I came on the scene at that point and implemented every safeguard I could find, but the concern still lingers.

Finally, dynamic URLs are not user friendly ... human beings generally do not know what to make of long strings of obscure parameters.

Apache has a feature called "mod_rewrite" that allows you to specify, via regex, that you want incoming URLs to be transformed in some way. Apache's instructions on this subject are here: URL Rewriting Guide. I have used that facility at Philadelphia Reflections to use static URLs for public use while still allowing me to use parameters to drive the website with the database.

Two excellent articles on this got me started:

Here's what I did:

Step 1: htaccess

I added these lines to the htaccess file

Options +FollowSymLinks
RewriteEngine on
RewriteRule ^(blog|topic)/([0-9]+)\.html?$ reflections.php?type=$1&key=$2

^(blog|topic)/([0-9]+)\.html?$ is the pattern to be compared against all incoming URLs.

If matched, it changes the URL to reflections.php?type=$1&key=$2
The ^ ... $ sequence in the pattern says that we will match the whole string, not just some part in the middle
(blog|topic)/ matches either "blog" or "topic" followed by "/"
([0-9]+) matches one or more digits
\.htm matches ".htm"
l? matches 0 or 1 lower-case Ls (so that we will match either htm or html)
In the replacement string $1 is replaced with the contents of the first () in the pattern: either "blog" or "topic"
... and $2 is replaced by the second () in the pattern, namely the numeric ID on the database of the blog or topic

The result is that

http://www.philadelphia-reflections.com/blog/906.htm

is transformed into

http://www.philadelphia-reflections.com/reflections.php?type=blog&key=906

The latter is what is passed in to me in the reflections.php routine, which tells me to pull up blog #906 from the database.

Both of these URLs are equivalent to the old, ugly dynamic URL

http://www.philadelphia-reflections.com/reflections.php?content=blogs_alpha/zmadame_butterfly.html

which still works, in case there are any legacy bookmarks or links out there, but going forward the new, simple, static URL is the face we will present to the world.

Step 2. SMOP

After the htaccess regex was debugged, all that was left was a simple matter of programming. In fact, I had to completely rewrite the driver script, reflections.php, and the XML creation script which creates the RSS, sitemap, etc. files; plus a lot more besides. It was a lot of work but the breakthrough was in figuring out the htaccess trick; everything else was just work.

Floating Three-Column CSS Layout

A current fad in web page styling is to use CSS exclusively to define the basic page sections. The "old" way of doing this was to use tables, but that's no longer stylish. Instead, we are exhorted to use CSS exclusively.

A very common page layout has a head and a foot with three columns sandwiched in between. Philadelphia Reflections uses this layout.

Most descriptions of this layout style that I have found Googling around the Internet involve absolute positioning which very often does not adapt well to differing screen sizes and browser window sizes. What we use here makes use of floating columns, which re-size themselves very nicely.

Several anomalies and quirks should be noted:

Each element is defined as a DIV
The left, right and center DIVs must be enclosed in a "wrapper" DIV
The three columns must be followed by a clear:both DIV
The center column must be below the left and right columns
The center column actually is as wide as the whole page (try including border-style:solid)

These quirks and anaomalies make me think that maybe this either isn't quite kosher or else may be superceded by later CSS definitions. But for the time being, this works very happily and both the HTML and the CSS validate perfectly well.

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN">
<html>
  <head>
    <title> Floating Three-column CSS example</title>

    <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">

<style type="text/css">


 #head {
  background-color:blue;
  color:white;
  text-align:center;
  }

 #wrap {
  }

 #left {
  float:left;
  width: 30%;
  }

 #right {
  float:right;
  width:30%;
  }

 #center {
  }

 #clear {
  clear:both;
  }

 #foot {
  background-color:red;
  color:white;
  text-align:center;
  }

</style>
  </head>

  <body>

    <div id="head">
      <p>Head</p>
    </div>

    <div id="wrap">

      <div id="left">
        <p>Left</p>
      </div>

      <div id="right">
        <p>Right</p>
      </div>

      <div id="center">
        <p>Center</p>
      </div>

      <div id="clear"></div>

    </div>

    <div id="foot">
      <p>Foot</p>
    </div>

  </body>
</html>

Ampersand Madness: Convert & to & to prevent XHTML errors

The whole subject of "encoding" gives me a headache.

Encoding In General

The first thing you have to know is: what is HTML encoding ... so look here:
⇒ http://htmlhelp.com/reference/html40/entities/
or here:
⇒ http://www.cookwood.com/html/extras/entities.html

(These are HTML encodings; URL encoding is something else again ... look here:
» http://www.blooberry.com/indexdot/html/topics/urlencoding.htm)

Ampersand Encoding and Conversion

Later on, you'll find out that the ampersand is a huge source of XHTML errors because it has to be written

&
or
&
or
&

but you will struggle endlessly with how to get the darn thing to stay converted. First of all, content providers feel justifiably justified in including bare naked "&"s wherever they please; second of all, you will find that encoded ampersands get stripped back to their bare naked selves by browsers and other well-meaning sorts.

So, my undying thanks to Michael Ash's Regex Blog for providing the regex pattern in the following bit of PHP code:


$pattern = '/&(?!(?i:\#((x([\dA-F]){1,5})|(104857[0-5]|10485[0-6]\d|1048[0-4]\d\d|104[0-7]\d{3}|10[0-3]\d{4}|0?\d{1,6}))|([A-Za-z\d.]{2,31}));)/i';
				
$replacement = '&amp;';
				
$string = preg_replace ( $pattern, $replacement, $string);

I don't know how it can possibly work, and I may yet eat my words, but for the moment it seems to do the trick.

≈

Ampersand Encoding In RSS

Another thing: & is the only ampersand encoding form acceptable to both RSS and Atom. So, look at the souce of this page and you will find that I use this encoding in the title ... that's because the title goes into the Title field of my RSS and Atom feeds.

≈

Webpage Printing

This site offers a Print button for all Reflections and Topics. Formatting the text on the pages to print nicely works quite well; how to specify what to do with images remains a bit unclear (as of August 2006).

The "trick", if it can be called that, to special print formatting is the media attribute for CSS styling. The main stylesheet for this website is called in a LINK statement as follows:

<link rel="stylesheet" type="text/css" media="all" href="stylesheets/reflectionsLayout.css">

The media attribute tells the browser to use this sylesheet for all media types, i.e., for screens and printers. In the pages that are formattted to print is a stylesheet that cascades below the main stylesheet and therefore supercedes it. This stylesheet controls the printing.

The specification of

<body onload="window.print()"> (all lower case for XHTML purposes)

is what forces the print dialog to appear.

The remaining problem is how to specify CSS formatting for images so that text flows around them as we want. The formatting seems to work on screen for all browsers but only on some browsers for printing.


<style type="text/css" media="print">

/*
	Style Sheet to print Philadelphia Reflections
*/


body			{
				margin:		0;
				padding:	0;
				width:		100%;
				}				
				

#wrapper		{
				margin:		0;
				padding:	0;
				width:		100%;
				}
		
		
#center			{
				margin-top:		0;
				margin-bottom:	0;
				margin-right:	0;
				margin-left:	0;
				padding:		0;
				}


				
#content		{
				font-size:120%;
				}
				
</style>

(my thanks to http://centricle.com/tools/html-entities/ for HTML encoding)

Editing HTML with PHP scripts

When creating scripts that allow a user to edit HTML, you have to ensure that the browser doesn't confuse the input with HTML to be rendered. I struggled with this long and hard and throughout the utilities section of this website are various hacks that I created with brute force. They work, but they are mostly ugly and all were time consuming.

Well, guess what? The PHP manual has a section on this subject and the solution is really rather elegant. Chaper 56. PHP and HTML. It's worth reading, but the essential bits are reproduced below:


Example 56-1. A hidden HTML form element
<?php
   echo "<input type='hidden' value='" . htmlspecialchars($data) . "' />\n";
?> 

Example 56-2. Data to be edited by the user
<?php
   echo "<textarea name='mydata'>\n";
   echo htmlspecialchars($data)."\n";
   echo "</textarea>";
?> 


Example 56-3. In a URL
<?php
   echo "<a href='" . htmlspecialchars("/nextpage.php?stage=23&data=" .
       urlencode($data)) . "'>\n";
?>

Open a new window with XHTML

Once upon a time you could say

<a href="link" target="_blank">Click to open a new window</a>

and a new window would open. Highly annoying if used very often, but sometimes it's the right thing to do.

And then XHTML comes along and this is not longer legal. target="_blank" is "deprecated" without a single word as to what a poor developer is to substitute.

Here's what you do:

<a href="link" onclick="window.open(this.href); return false;">Click to open a new window</a>

DHTML, PHP and MySQL References

Start with

DHTML and CSS for the World Wide Web:
Visual QuickStart Guide
by Jason Cranford Teague

DHTML = HTML + CSS + DOM + JavaScript

This book will get you up and running quickly with client-side programming.

Then you need to learn server-side programming.

PHP is an open source server-side scripting language that is easy to learn and very powerful. MySQL is the same ... open source relational database.

The text book for these technologies is

PHP and MySQL for Dynamic Web Sites:
Visual QuickPro Guide
by Larry Ullman

Master those two books and you'll be creating very powerful scripts on both the client and server side that produce dynamic and elegant results.

While you're in the process of doing this, you will constantly need to reference manuals for syntax, functions, etc. There are many, but two will suffice for 90% of what you need:

PHP Manual

W3 Schools

Regular Expressions

Anyone who has used the expression *.doc to search for Word files has used Regular Expressions ("regex") without realizing it. Regex arose from mathematical theory and is available in many programming languages; it is simply the only way to deal with large amounts of text. And yet most people are completely unaware of it.

Philadelphia Reflections uses regex extensively for two primary purposes: (1) checking input from forms and (2) modifying HTML input in during the creation of articles for the site.

The text PHP and MySQL by Larry Ullman has a very good introduction to regex in his chapter on security.

The great advantage of regex is that it can identify very complex patterns in a mass of text. The great disadvantage of regex is that it has developed in sort of an underground way and there exist numerous varieties that are essentially incompatible. PHP offers two regex functions: one for the POSIX Extended variety of regex and he other for the Perl language compatible vesion called PCRE. POSIX is less powerful but far easier to learn. JavaScript offers its own variety of regex which isn't quite the same as either of the two PHP versions.

References include the Ullman book, the PHP online manual has a number of handy tips on regex use in its two supported varieties, the O'Reilly book Mastering Regular Expressions is interesting and Jan Goyvaerts has a very helpful website (http://www.regular-expressions.info/) and book Regular Expressions: The Complete Tutorial.

My experience is that this area requires diligent hacking which may be sub optimal but unavoidable ... for this purpose, Jan Goyvaerts' Regex Buddy is indispensible; you simply must get this program if you hope to make anything of Regex.

Here are examples of checking for a valid email address in both Javascript and PHP:

Javascript


// check email

var namePattern = /^[a-zA-Z0-9][a-z0-9_.-]*@[a-z0-9.-]+\.[a-z]{2,4}$/i;

document.comment_form.email.value = trim(document.comment_form.email.value);
					
if	(
	(  document.comment_form.email.value.length > 0)
			&&
	(! document.comment_form.email.value == "[none]")
			&&
	(! document.comment_form.email.value.match(namePattern))
	) 
		{
			alert("Please enter a valid email address");
    			document.comment_form.email.focus();
       			document.comment_form.email.select();
			problem = "yes";
       			return false;
   		}

PHP


// check email

$emailpattern = "^[a-zA-Z0-9][a-z0-9_.-]*@[a-z0-9.-]+\.[a-z]{2,4}$";
		
if	(
	(trim(strlen($_POST['email']))  > 0)
		
			and
					
	(!$_POST['email'] == "[none]")
			
			and

	(!eregi ($emailpattern, stripslashes(trim($_POST['email']))))
	)
		{
		$inputerror		=	TRUE;
		$inputerrormessage	.=	"<br />* An invalid email address was entered";
		}

Incomprehensible? Yes, absolutely.

Useful? More than you can realize until you are actually faced with the problem of, say, verifying that a user has input a valid email format, or trying to figure out whether a user-input IMG tag is using the correct syntax; or else maybe trying to convert a huge web page from XHTML 1.1 to HTML 4.01 because you've determined that the browser is syntactically crippled.

And, once you get deep into it, the stuff is actually intriguing and fun.

Javascript: document.write and XHTML

For reasons that make no sense to me, the Javascript command document.write does not work when your page is rendered properly in XHMTL (as described elsewhere in this Topic).

I have searched the web in vain to find a Javascript solution. Many are offered but none work worth a damn.

So don't bother. Use PHP's echo function. It works perfectly.

Captcha

With the rise of spam entries in web forms, a security feature called "captcha" has been developed.

CAPTCHA stands for "Completely Automated Public Turing test to tell Computers and Humans Apart". The idea is that only a human could read the letters contained in the image and then enter them in the form. "Accessibility", ie., designing websites to accommodate people with handicaps is obviously hindered by Captcha; but at least given our experience with this website, spamming is a huge problem and the inability of handicapped people to leave comments is a price we are willing to pay to rid ourselves of spam. The W3C, the Godhead of web standards, does not agree with me and lectures at length on the futility of captcha: Inaccessibility of CAPTCHA. Whatever. I may get around to implementing some of their recommendations later, if we continue to be spammed.

Spammers have countered captcha in a number of ways. The first is OCR, which is why the images have fuzzy backgrounds and distorted letters: trying to defeat OCR programs. As OCR techniques have improved, captcha programs have moved from letters to "objects" such as kittens, boxes, etc., which are thought to be harder for computers to recognize; harder for people, too: cat vs kitten, for example. I am amazed to learn during my captcha research that there are spammers who offer micro-payments to people in India, etc. to enter hundreds of spam manually in captcha-ed websites that have defeated their automated spamming systems. Move, counter move; seemingly endlessly.

In this website captcha has been implemented using PHP: the comments form that appears at the end of every page has an image created using the PHP image-creation routines which has random characters in it. If the characters in the image are entered correctly in the form, the comments are entered into the database.

I cribbed the PHP captcha code from http://www.white-hat-web-design.co.uk/articles/php-captcha.php and it worked right out of the box with the minor exception that the form HTML didn't quite pass XHTML muster; easily fixed. (I have subsequently discovered that PHP security and sessions don't play well together; this problem remains unresolved and I've had to turn off captcha processing for my secure pages.)

I implemented a number of other spam counter measures before I got around to captcha, which involved noticing what the spammers did and writing code to frustrate it. I am constantly on the lookout for new security techniques to implement.

RSS, Atom, Syndication, etc.

The world is full of XML and XML-like file formats for syndication purposes

Here's the list of files we generate automatically for submission to search engines and such.

(For right now, things are a bit abbreviated)

Ping Blogroller manually
Ping MyYahoo manually

The RSS and Atom validator (http://feedvalidator.org/) has a length restriction. I don't know what it is, exactly, but it bombs if your file is "too long". Since most syndication readers run the validator before they'll accept a feed, I have resorted to creating a short file, which is what I point to in my meta tags.

Here's how I provide change frequency and priority for our Google sitemap (in PHP ... $mod is the variable containing the date last modified)

$GOOGLEpriority = "0.0"; $GOOGLEfreq = "yearly";	// default

if ($mod > mktime(0,0,0) - 86400*210)	{$GOOGLEpriority = "0.1"; $GOOGLEfreq = "monthly";}	// past 210 days
if ($mod > mktime(0,0,0) - 86400*180)	{$GOOGLEpriority = "0.2"; $GOOGLEfreq = "monthly";}	// past 180 days
if ($mod > mktime(0,0,0) - 86400*150)	{$GOOGLEpriority = "0.3"; $GOOGLEfreq = "monthly";}	// past 150 days
if ($mod > mktime(0,0,0) - 86400*120)	{$GOOGLEpriority = "0.4"; $GOOGLEfreq = "monthly";}	// past 120 days
if ($mod > mktime(0,0,0) - 86400*90)	{$GOOGLEpriority = "0.5"; $GOOGLEfreq = "monthly";}	// past 90 days
if ($mod > mktime(0,0,0) - 86400*60)	{$GOOGLEpriority = "0.6"; $GOOGLEfreq = "monthly";}	// past 60 days
if ($mod > mktime(0,0,0) - 86400*30)	{$GOOGLEpriority = "0.7"; $GOOGLEfreq = "monthly";}	// past 30 days
if ($mod > mktime(0,0,0) - 86400*7)	{$GOOGLEpriority = "0.8"; $GOOGLEfreq = "weekly";}	// past 7 days
if ($mod > mktime(0,0,0) - 86400)	{$GOOGLEpriority = "0.9"; $GOOGLEfreq = "daily";}	// yesterday
if ($GOOGLEmoddate == date("Y-m-d"))	{$GOOGLEpriority = "1.0"; $GOOGLEfreq = "hourly";}	// today

IDIF is a stupid format: it includes the entire blog_contents, so the files are huge. In the process of setting this up, I learned that flat files have a maximum size of 1.4 megs or so (the size of an old floppy disk), so I had to create more than one.

Which explains the stupid concept of a "pointer file"; instead of just giving Yahoo the IDIF file itself, you give it a pointer file with URLs pointing to the multitude of IDIF files. Really stupid.

News flash, after finding the Journal Of Ovid on the web, I learned about length restrictions for the input fields (described below). This information was not contained on the Yahoo web site describing their file format. It considerably reduced the file sizes but I retained the structure of multiple files because who knows what I'll learn next?

IDIF title must be a maximum of 80 characters
IDIF description must be a maximum of 180 characters
IDIF body must be a maximum of 1000 characters
I'm only guessing about keywords

Thanks to the Journal Of Ovid on the web for this secret information

From the inside out: trim, replace whitespace (thanks to the PHP manual for this), shorten to maximum length

$IDIFtitle		= substr( preg_replace ('/\s\s+/', ' ', trim($title) ), 0, 80 );
$IDIFdescription	= substr( preg_replace ('/\s\s+/', ' ', trim($description) ), 0, 180 );
$IDIFkeywords		= substr( preg_replace ('/\s\s+/', ' ', trim($keywords) ), 0, 79 ) . " ";
$IDIFblog_contents	= substr( preg_replace ('/\s\s+/', ' ', trim($blog_contents) ), 0, 1000 );

Yahoo is said to support a simple text file list of URLs "urllist.txt" Documentation, of course, is scarce

(my thanks to http://centricle.com/tools/html-entities/ for HTML encoding)

Web Standards Validation

There are two primary aspects of a website that need validation:

1. (X)HTML

You can use the W3C's QA Markup Validation Service.
The URL to test the main page of Philadelphia Reflections is http://validator.w3.org/

Firefox has several useful add-ons for (X)HTML validation; one that uses Tidy is here: Html Validator

2. CSS

The W3C has a validation service for CSS, too.
For Philadelphia Reflections, the following URL checks all the CSS definitions in the main page: http://jigsaw.w3.org/css-validator/ (note: this validator is a little flakey: it produces different answers for the same file; you have to refresh a couple of times to get the whole story)

Firefox has several useful web developer add-on tools; try this one: Web Developer

Once you've gotten the HTML and CSS basics under control, there are other aspects of your site that you will want to validate:

Broken Links

The W3C will check all your links for both response time and validity.
http://validator.w3.org/checklink/checklink

Tidy

There is an absolutely lovely program called HTML Tidy, origianlly written by Dave Raggett and decribed by the W3C here: http://www.w3.org/People/Raggett/tidy/

Calls to Tidy are available in some newer renditions of PHP (sadly, not the one we are using), however, on Widows (only) versions of Firefox and Mozilla, you can download an extension that will provide all the Tidy functions in your browser! ... https://addons.mozilla.org/firefox/249/. This a fantastic feature that I use all the time.

Syndication XML Validation

Validating RSS and Atom files is greatly facilitated by http://feedvalidator.org/. It has a number of quirks, the worst of which is that it has a length limitation that we exceed and so we have to provide "short" syndication files since all the feed aggregators use this facility and reject any feeds that aren't validated by it.

Google Sitemap Validation

If you submit a sitemap to Google through their Webmaster Tools facility they will validate your sitemap when they load it. An external validation tool is available here: Validome Google Sitemap(s) Validator

Yahoo and Microsoft have agreed to support Google's Sitemap protocol and to support the inclusion of the line "Sitemap: http://www.philadelphia-reflections.com/sitemap.xml" in robots.txt. If other search engines adopt this facility it will make it much easier to get into the world's many search engines ... they'll pick up this line instead of us having to hunt them down.

Meta Tag Validator

As you puzzle the mysteries of search-engine indexing, you'll want to check your meta tags: http://www.widexl.com/remote/search-engines/metatag-analyzer.html

gzip Compression & Headers

When you start getting really fancy and want to include automatic gzip compression, you'll want to see it in action and you'll want to check out all of your HTTP headers: http://www.gidnetwork.com/tools/gzip-test.php

Response Time

Of course, the reason you''re experimenting with gzip is because you're concerned about response time.
(1) Try this site for a detailed analysis: http://www.websitepulse.com/help/tools.php
(2) Firefox to the rescue again: FasterFox is another lovely add-in: https://addons.mozilla.org/firefox/1269/

Geo Tags

Check the validity of your geo tags here: Geo-Tag Validator

Big List

http://uitest.com/en/analysis/ is the mother of all lists of validations routines

CSS Zen Garden Suggestions

Here is a list of links (that open in their own pages) that show some of my favorite web designs. The CSS Zen Garden is a website that illustrates what can be done with clever CSS design. The HTML and the content are exactly the same in each of these links, only the CSS changes; but what a difference!

Another website to consider is http://www.freecsstemplates.org/

Font Families

The following link shows the results of a survey done to find out which font families are installed on Windows machines. This should help determine which fonts to use.

http://www.codestyle.org/css/font-family/sampler-WindowsResults.shtml

Identifont is a site that helps identify good font choices.

Web hosting providers

Choosing a web hosting service provider is difficult. There's no brand to rely on and an Internet search turns up confusing claims and offers. We have used two providers:

Internet Planners is the service currently running this web site. They have provided what they promised. They do not run the latest versions of MySQL or PHP which means certain newer features are not available but it probably improves their stability and what they offer is, in fact, provided. What was missing that we noticed included
- RSS parsing (Magpie was a good substitute)
- and HTML Tidy, which we've just lived without, substituting our own Regular Expressions.
Network Solutions, on the other hand, has provided very poor service and has high fees. They run later versions of PHP and MySQL than Internet Planners, but their implementation is poor and the result is that critical functions are not available: user authentication for example (how can an Internet service provider not support user authentication?).

Based on our experience, Internet Planners is a reasonable choice for web hosting; Network Solutions is a bad choice.

Modern Printing and Post-Modern Printing

There are now many more authors than there are publishers of books. A strong force driving this disparity is that just about every school child owns and uses a home computer, but commercial printing presses have difficulty accepting copy from that source, especially photographs. The inherent mismatch may have something to do with commercial printing presses switching from Gutenberg's movable type to printing page images at least a generation or so ago, while computer printers took the direction of perfecting the Gutenberg method rather than replacing it. Let's give a simplified explanation.

Very few commercial printers do physical type-setting anymore. Since 1993 when Adobe invented the method, they generally work from what amounts to a photograph of each page, called a PDF or portable description format. To some extent, portions of a page can be stitched together like a patchwork quilt, but all of the pieces must agree on certain basic rules, one of which is that everything is printed at 300 dots per inch. That's essentially 300 pixels per inch.

By contrast, low-volume desktop printers can afford to take the time to examine each character or image as it comes along and readjust as appropriate. Essentially, computer printers do individual typesetting every time a new page is printed. They are thus able to exploit considerable compression for storage, or for the speed of electronic transmission. For them,72 dots per inch are sufficient, since computer-driven printers have acquired the facility to guess the gaps between dots well enough to fool the eye of the reader. Since the same thing is true of display monitors as computer printers, there is considerable resistance to bridging the remaining gap to the needs of the mass-printing industry.

Unfortunately, it is hard to convert photo images from 72 to 300 dpi. The best illustration so far offered is the washing of wool socks. You can throw argyle socks in a washing machine and they will shrink to the size of baby booties, but they won't stretch back up if you decide to wear them. The usual method attempted is to enlarge a small picture and take a second picture of it, then enlarge the enlargement, and so on. The Genuine Fractals program by Altamira Group comes closest to achieving the desired result, but even it has limits. When someone has more than a very few pictures that need stretching between Internet screen display and commercial printers, the best available advice is to store two different-density copies of the same image and use each as required.

Slight re-design of work flow is advised, to include a third stored version of the image. The biggest possible image with the most pixels possible is stored on the author or publisher's own computer. A second, shriveled 72-dpi, image is sent to be stored on the host computer of a website, and used for desktop printing as well. When commercial printing is desired, the large stored image can be shriveled to 300 dpi and this third version of the image is incorporated into Office Word for editing, subsequently incorporated into a pdf file for final printing. Fine art display or other highly demanding graphics will follow this general outline, as well. Those who have any aspiration for high-density output would be well advised to go back to the original photograph. In 2008, that translates into the maxim: Always take all your photos in RAW format for permanent storage. Less demanding output can be produced from degraded copies of the original, the most common of which is now the so-called JPEG format. Those who discard the original RAW image, are almost always sorry.

Server-Side gzip Compression

Compression can reduce the size of the text (not images) of your web pages as they are transmitted outbound to the client. This will have only a small impact on response time over modern fiber connections but it will significantly reduce your bandwidth consumption (70% on average on this site.)

In XHTML vs. HTML I show how I implemented gzip compression on this site. The problem with that method is that it's a pain. So on another website I tried out the Apache htaccess method to instruct the server to compress all outbound pages. Works like a charm.


# See http://httpd.apache.org/docs/2.0/mod/mod_deflate.html

# Insert filter
SetOutputFilter DEFLATE

# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html

# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip

# MSIE masquerades as Netscape, but it is fine
# BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

# NOTE: Due to a bug in mod_setenvif up to Apache 2.0.48
# the above regex won't work. You can use the following
# workaround to get the desired effect:
BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html

# Don't compress images
SetEnvIfNoCase Request_URI \
\.(?:gif|jpe?g|png)$ no-gzip dont-vary

# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary

SQL To Exclude A List Of Items

Let's say you have a table "Primary" that contains an "Email" field.

You would like to select all the email addresses in Primary except for the list of email addresses in the Email field in a table "Exclude".

This SQL will exclude the emails in "Primary" based on those contained in "Exclude".

SELECT * FROM Primary WHERE ((Primary.Email) Not In (SELECT Email FROM Exclude))

HTML Anchor without an HREF?

Sometimes I want to execute a JavaScript function when a user clicks a link, but nothing else.

If I omit the href entirely, the cursor doesn't change and some browsers don't recognize the text as a link:

<a onclick="function();">

If I include the pound sign, which seems to be a very poopular trick, I get sent to the top of the current page, which messes up both History and the backspace button; to say nothing of the fact that I don't want to jump to the top of the page:

<a href="#" onclick="function();">

Including the function in the href and omitting the onclick seems to be the answer to my specific problem:

<a href="javascript: function();">

Once again, my thanks to http://centricle.com/tools/html-entities/ for HTML encoding.