> I'm looking at my default memory_limit of 16M, and that seems small.
>

Assuming that's 16MB per simultaneous PHP request, that means you can
service less than 200 simultaneous requests on a 3GB server. If the
PHP process only needs to make database calls and process a 3kB string
(webpage), then 16MB doesn't sound so small.

> Is adding more memory to the PHP processes helpful at all in terms of PHP
> execution speed (assuming a "loaded" site)?
>

I haven't tuned PHP programs before either, but I would guess that
it's one of three cases:
1) If there's not enough memory per process, then the extra memory
requirements go to swap (agonizingly slow because we're talking about
very short lived processes).
2) If there's not enough memory per process, then any process that
goes over its requirements aborts and sends a default static error
page.
3) The limit is a minimum, not a maximum. Threads that need more will
take a bit of extra time to request an additional memory allocation.

Option 1 would indicate that PHP is very poorly designed, and I think
that is unlikely.

Option 2 might make sense from a security standpoint, if you would
prefer for a poorly written PHP script to just die instead of hogging
all of your system's resources.

Option 3 is what I would prefer as a developer (and this is how I
would guess it is designed), because then an incorrect value of
memory_limit would only have a slight performance penalty that I could
ignore until my server had performance issues. In this case, you would
want memory_limit to be higher than what 90% of your scripts would use
and lower or not much higher than what 100% of them use. Putting
memory_limit much higher than that would waste memory without speeding
anything up. There should be a memory profiling tool that would help
to determine if the setting needs to be changed.