We are facing a peculiar problem. I'm providing as much details as I can. Even so, tell me if I should provide more info.
LAMP server before install - RHEL 3 with Apache 2, PHP 4, MySQL 4. After install - RHEL 5, Apache 2.2.3, PHP 5.1.6, MySQL 5.0.2
The problem we are facing is the site performing badly when accessed from certain locations - from our corporate LAN, which is behind Squid. Certain colleagues in other companies (we suspect behind they are also behind Squid or some proxy) also reported with gemeric messages like "site isn't loading".
Some symptoms are: 1. Images, CSS, JS files not loading or loading incompletely.
2. If we view source, we see the HTML is fully downloaded, but not rendered.
2. PHP pages being asked to be saved (the saved pages are blank) This problem is prominently noticeable in PHPMyAdmin.
3. The performance improves 9may be illusion) over time, but any new drastic changes brings it to square one.
4. Awstats page takes forever to load comletely.
5. Sometimes the sites are inaccessible in toto.
6. Files not downloading through FTP (even 200K fails).
The perplexing thing is I and my colleague have accessed the server from home (Triband and cable net respectively) and everything works! We don't know what the problem is, where it is, etc.
Any ideas?
rohit,
It is very difficult to tell you exactly what the problem is without looking into it.
Is squid the only difference between users in your company and your place?
From the problems you are talking about, the http headers that are coming
from squid are not set the way they are set from your application server.
Just do a diff between the headers with and without squid for the same page to do further analysis on the problem.
Please look into the squid configuration that is in place and do the necessary changes.
On 10/31/07, rohit bhute rvbhute@gmail.com wrote:
We are facing a peculiar problem. I'm providing as much details as I can. Even so, tell me if I should provide more info.
LAMP server before install - RHEL 3 with Apache 2, PHP 4, MySQL 4. After install - RHEL 5, Apache 2.2.3, PHP 5.1.6, MySQL 5.0.2
The problem we are facing is the site performing badly when accessed from certain locations - from our corporate LAN, which is behind Squid. Certain colleagues in other companies (we suspect behind they are also behind Squid or some proxy) also reported with gemeric messages like "site isn't loading".
Some symptoms are:
Images, CSS, JS files not loading or loading incompletely.
If we view source, we see the HTML is fully downloaded, but not
rendered.
- PHP pages being asked to be saved (the saved pages are blank)
This problem is prominently noticeable in PHPMyAdmin.
- The performance improves 9may be illusion) over time, but any new
drastic changes brings it to square one.
Awstats page takes forever to load comletely.
Sometimes the sites are inaccessible in toto.
Files not downloading through FTP (even 200K fails).
The perplexing thing is I and my colleague have accessed the server from home (Triband and cable net respectively) and everything works! We don't know what the problem is, where it is, etc.
Any ideas?
-- Rohit V. Bhute http://rvbhute.org/
On 10/31/07, PV Sundarram pvsundarram@gmail.com wrote:
Is squid the only difference between users in your company and your place?
Yes.
However, before I take it upstairs (for the admins to start fiddling with Squid - I'm just the junior web dev), there are some points I feel that take Squid out of the picture and ask me to look at the LAMP server.
1. The Squid config hasn't changed (that I know of) from before the problem occurred to now - it hasn't been changed in years. Period.
2. Other sites are "working fine".
3. This wasn't occurring before the upgrade.
I'm at work right now. Will take a look at the headers as you suggested.
Is there anything particular to look out for on the LAMP server machine itself in the mean while?
On 10/31/07, rohit bhute rvbhute@gmail.com wrote:
Is there anything particular to look out for on the LAMP server machine itself in the mean while?
Have a look at the files you are getting for saving while using phpmyadmin. Save the file and look at what contents you are getting. It will either be the php's source file or the processed html. Depending on what you get, you will know the source of the problem.
On Wed, 2007-10-31 at 15:13 +0530, rohit bhute wrote:
Yes.
However, before I take it upstairs (for the admins to start fiddling with Squid - I'm just the junior web dev), there are some points I feel that take Squid out of the picture and ask me to look at the LAMP server.
tsk tsk. okay, listen up kiddies :P ( no offense meant ), in the sysadmin business theres one thing that EVERYONE should REMEMBER. It's like a directive from God :P
"Don't fix it if it ain't broken!"
Repeat it several times and _dont_ do this again.
Lets look at the facts:
Fact 1: Old Server ran PHP4, MySQL4
Fact 2: New server runs PHP5, MySQL5
Fact 3: App was probably coded in the PHP4 environment ( this is actually an inference that I'm making out of your statements )
Fact 4: See [1]
Fact 5: PHP5 breaks many PHP4 apps :P
Fact 6: MySQL5 also is responsible for breaking stuff coded for MySQL4 ( please read the release notes for further info ;) )
I hope you can understand what I'm getting at. Your app is probably behaving weirdly because of the incompatibilities between PHP4 and PHP5. I suspect MySQL ain't the source of the problem.
Now, a personal note. Take this as constructive criticism. Next time you guys get the itch of upgrading stuff, please set up a box and test your upgrades. If not that, atleast read the release notes of the softwares whose major version numbers are changing ( PHP, MySQL, RHEL in this case ). It'll give you a clue as to what you might expect. I really feel you should downgrade to your older configuration. I seriously dont understand why everyone wants bleeding edge stuff? RHEL3 is properly supported by Redhat. If you can't afford the support, install CentOS.
Geez what were your seniors thinking??? :O
[1] ->
- The Squid config hasn't changed (that I know of) from before the
problem occurred to now - it hasn't been changed in years. Period.
Other sites are "working fine".
This wasn't occurring before the upgrade.
On Fri, 02 Nov 2007 23:23:41 +0530 Dinesh wrote:
Fact 3: App was probably coded in the PHP4 environment ( this is actually an inference that I'm making out of your statements )
Fact 4: See [1]
Fact 5: PHP5 breaks many PHP4 apps :P
Fact 6: MySQL5 also is responsible for breaking stuff coded for MySQL4 ( please read the release notes for further info ;) )
I hope you can understand what I'm getting at. Your app is probably behaving weirdly because of the incompatibilities between PHP4 and PHP5. I suspect MySQL ain't the source of the problem.
If I remember right, he said the same is working fine from outside.
OK, I've manged to confuse the problem. So:
1. The 'apps' are basic run-of-the-mill websites. Nothing fancy. 2. We tested it on a local machine with RHEL 5/PHP 5/MySQL 5. 3. The php.ini and httpd.conf files were suitably edited. 4. Yes, *it does work without a glitch* from my connection (Triband) and my colleague's connection (cable net - Fivenet - Kurla). I have no logical explanation. 5. My colleague is the only senior and he comes from an ASP/IIS background. On this, we are equals.
The problem seems to be (in layman's term) 'catching a file from the filesystem'. And it was detected on the local machine, but we didn't anticipate it to blow up like this.
When we upload a new website module to the server, then the problem shows up - page taking a long time to load, etc. We go home and check - its working fine. And its working fine when seen from office by the second session the next day.
In fact, PHPMyAdmin has also stabilized mostly. So has Awstats.
We face similar problem in FTP - I tried to download a SQL dump file - the speed was abysmal and the download had to be aborted.
I compared the response headers (www.redboxrewards.com).
From my home and office respectively:
**** Date Wed, 31 Oct 2007 15:04:59 GMT Server Apache/2.2.3 (Red Hat) X-Powered-By PHP/5.1.6 Set-Cookie PHPSESSID=gd13u9u09n9cehnb951m8v1ok2; path=/ Expires Thu, 19 Nov 1981 08:52:00 GMT Cache-Control no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma no-cache Content-Encoding gzip Vary Accept-Encoding Content-Length 3314 Keep-Alive timeout=30, max=1000 Connection Keep-Alive Content-Type text/html; charset=UTF-8 **** Date Mon, 05 Nov 2007 07:06:57 GMT Server Apache/2.2.3 (Red Hat) X-Powered-By PHP/5.1.6 Set-Cookie PHPSESSID=trlhc2cnh0pvtng35idgj0ek60; path=/ Expires Thu, 19 Nov 1981 08:52:00 GMT Cache-Control no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma no-cache Vary Accept-Encoding Content-Length 17153 Content-Type text/html; charset=UTF-8 X-Cache MISS from sigma X-Cache-Lookup MISS from sigma:3128 Connection keep-alive **** It looks like the the Squid proxy isn't asking for gzipped data.
Quoting rohit bhute rvbhute@gmail.com:
We are facing a peculiar problem. I'm providing as much details as I can. Even so, tell me if I should provide more info.
LAMP server before install - RHEL 3 with Apache 2, PHP 4, MySQL 4. After install - RHEL 5, Apache 2.2.3, PHP 5.1.6, MySQL 5.0.2
Did you upgrade the LAMP server, or did you migrate your app to the said RHEL5 server?
Some symptoms are:
Images, CSS, JS files not loading or loading incompletely.
If we view source, we see the HTML is fully downloaded, but not rendered.
PHP pages being asked to be saved (the saved pages are blank)
This problem is prominently noticeable in PHPMyAdmin.
Looks like the PHP process is getting killed for some reason. If the LAMP configuration was in good shape, and was serving PHP apps before, then have a look at apache log files for any symptoms.
And just to be sure, make sure that the LAMP server hardware(harddisk) isnt dying. When harddisk makes friends with Mr. Murphy, daemons behave strangely ;-)
Anurag
On 10/31/07, Anurag anurag@gnuer.org wrote:
Did you upgrade the LAMP server, or did you migrate your app to the said RHEL5 server?
Migrated the websites to RHEL 5. I simply took a backup, installed RHEL 5 and restored the apps. Bar some config changes in httpd.conf (from apache 2.0 to 2.2), no other modification were done.
Looks like the PHP process is getting killed for some reason. If the LAMP configuration was in good shape, and was serving PHP apps before, then have a look at apache log files for any symptoms.
This is the new LAMP stack, not even 30 days old. Will download the logs and go through them.
A point to note is when we were doing a tail on the log file (access and error is in one file; we are using virtual hosts), the HTTP status code is 200 even for files (images, CSS, JS) which are not loading on our terminals.
And just to be sure, make sure that the LAMP server hardware(harddisk) isnt dying. When harddisk makes friends with Mr. Murphy, daemons behave strangely ;-)
Its a brand new Dell server bought in March. Nevertheless, when Mr Murphy is involved... I'll take this up with the admins.
-
Anurag -
Dinesh Joshi -
Mehul Ved -
PV Sundarram -
rohit bhute