Tuesday, May 19, 2009

squid localhost

I installed Squid earlier this week to track all the http traffic.... effectively web scrapper. I stumbled on one issue that took more than 2 hours to debug...
http_access allow localhost

was needed so that my local machine traffic can be peroxided.

After the traffic started flowing, stumbled on issue that the 'dynamic' web content is not cached. While this can make sense as proxy, Squid is not usable for my use. I wish this feature was provided as charity at-least.

The data is binary - not plain text. This makes it difficult to parse easily...

I had to resort to Firefox extension slogger on firefox 2.0 (instead of the latest as of today 3.0.10). This combo did work and got my task done.

Blog Archive