Caching Youtube Using Squid Caching Proxy

I'm doing quick post today as I'm quite busy right now. But I don't want to make my friend Piju disappointed as he request me (quite so long... sorry heheh) about my changes on his previous squid.conf to cache youtube which is not working anymore since youtube make some changes on their video URL.

Here is my new /etc/squid/squid.conf

acl all src all
acl manager proto cache_object
acl localhost src 127.0.0.1/32
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32
acl localnet src 10.0.0.0/8 # RFC1918 possible internal network
acl localnet src 172.16.0.0/12  # RFC1918 possible internal network
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl SSL_ports port 443
acl Safe_ports port 80      # http
acl Safe_ports port 21      # ftp
acl Safe_ports port 443     # https
acl Safe_ports port 70      # gopher
acl Safe_ports port 210     # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280     # http-mgmt
acl Safe_ports port 488     # gss-http
acl Safe_ports port 591     # filemaker
acl Safe_ports port 777     # multiling http
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localnet
http_access deny all
icp_access allow localnet
icp_access deny all
http_port 31288
hierarchy_stoplist cgi-bin ?
cache_mem 2048 MB
maximum_object_size_in_memory 1024 KB
cache_dir ufs /disk2-cache/var/cache 150000 16 256
cache_dir ufs /disk1-1/squid-cache 150000 16 256
maximum_object_size 128 MB
access_log /disk2-cache/var/logs/access.log squid
cache_log /disk2-cache/var/logs/cache.log
cache_store_log /disk2-cache/var/logs/store.log
pid_filename /disk2-cache/var/logs/squid.pid
netdb_filename /disk2-cache/var/logs/netdb.state
storeurl_rewrite_children 50
refresh_pattern -i \.flv$          1440   80%    10080 ignore-no-cache override-expire ignore-private
refresh_pattern ^ftp:       1440    20% 10080 ignore-no-cache override-expire ignore-private
refresh_pattern ^http://[A-Za-z0-9]+\.lscache[0-9]\.c\.youtube\.com    9999999 90% 999999999 ignore-no-cache override-expire ignore-private
refresh_pattern ^http://[a-z0-9]+\.youtube\.com                        9999999 90% 999999999 ignore-no-cache override-expire ignore-private
refresh_pattern ^http://[a-z]+\.youtube\.com                           9999999 90% 999999999 ignore-no-cache override-expire ignore-private
refresh_pattern ^http://[a-z0-9]+\.ytimg\.com                          9999999 90% 999999999 ignore-no-cache override-expire ignore-private
refresh_pattern ^http://*\.youtube\.com     9999999  90%  999999999 ignore-no-cache override-expire ignore-private
refresh_pattern get_video\?video_id         9999999  90%  999999999 ignore-no-cache override-expire ignore-private
refresh_pattern youtube\.com/get_video\?    9999999  90%  999999999 ignore-no-cache override-expire ignore-private
refresh_pattern ^http://*.youtube.com/.*    9999999  100% 999999999 ignore-no-cache override-expire ignore-private
refresh_pattern (get_video\?|videoplayback\?|videodownload\?) 10080 99.99999% 999999 override-expire ignore-reload ignore-private negative-ttl=0
refresh_pattern ^gopher:    1440    0%  1440
refresh_pattern -i (/cgi-bin/|\?) 0 0%  0
refresh_pattern .               0       40%     4320
acl store_rewrite_list url_regex -i \.youtube\.com\/get_video\?
acl store_rewrite_list url_regex -i \.youtube\.com\/videoplayback\.youtube\.com\/videoplay \.youtube\.com\/get_video\?
acl store_rewrite_list url_regex -i \.youtube\.[a-z][a-z]\/videoplayback\.youtube\.[a-z][a-z]\/videoplay \.youtube\.[a-z][a-z]\/get_video\?
acl store_rewrite_list url_regex -i \.googlevideo\.com\/videoplayback\.googlevideo\.com\/videoplay \.googlevideo\.com\/get_video\?
acl store_rewrite_list url_regex -i \.google\.com\/videoplayback\.google\.com\/videoplay \.google\.com\/get_video\?
acl store_rewrite_list url_regex -i \.google\.[a-z][a-z]\/videoplayback\.google\.[a-z][a-z]\/videoplay \.google\.[a-z][a-z]\/get_video\?
acl store_rewrite_list url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/videoplayback\?
acl store_rewrite_list url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/videoplay\?
acl store_rewrite_list url_regex -i (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\/get_video\?
acl store_rewrite_list url_regex -i http://video\..*fbcdn\.net.*\.mp4.*
acl store_rewrite_list url_regex -i http://.[0-9]\.[0-9][0-9]\.channel\.facebook\.com/.*
acl store_rewrite_list url_regex -i http://.*\.mp4?
acl store_rewrite_list url_regex -i http://www\.facebook\.com/ajax/flash/.*
acl store_rewrite_list url_regex -i http://.*\.ak\.fbcdn\.net/.*
acl store_rewrite_list url_regex -i \.geo.yahoo\.com\?
storeurl_access allow store_rewrite_list
storeurl_access deny all
storeurl_rewrite_program /etc/squid/youtube
quick_abort_min 500 KB
acl shoutcast rep_header X-HTTP09-First-Line ^ICY.[0-9]
upgrade_http0.9 deny shoutcast
acl apache rep_header Server ^Apache
broken_vary_encoding allow apache
cache_mgr apogee@apogeek.com
cache_effective_user squid
cache_effective_group squid
snmp_port 3401
acl aclname snmp_community string
acl snmppublic snmp_community public
snmp_access allow snmppublic all
snmp_outgoing_address 0.0.0.0
dns_nameservers 8.8.8.8
dns_nameservers 8.8.4.4
dns_nameservers 4.2.2.2
coredump_dir /disk2-cache/var/cache

And here is the /etc/squid/youtube Perl script

#!/usr/bin/perl
$|=1;
while (<>) {
@X = split;
$url = $X[0];
if ($url=~s@^http://(.*?)/videoplayback\?(.*)id=(.*?)&.*@squid://videos.youtube.INTERNAL/ID=$3@){}
elsif
    ($url=~s@^http://(.*?)/videoplayback\?(.*)id=(.*?)@squid://videos.youtube.INTERNAL/ID=$3@){}
elsif
    ($url=~s@^http://(.*?)/videoplay\?(.*)id=(.*?)&.*@squid://videos.youtube.INTERNAL/ID=$3@){}
elsif
    ($url=~s@^http://(.*?)/videoplay\?(.*)id=(.*?)@squid://videos.youtube.INTERNAL/ID=$3@){}
elsif
    ($url=~s@^http://(.*?)/get_video\?(.*)video_id=(.*?)&.*@squid://videos.youtube.INTERNAL/ID=$3@){}
elsif
    ($url=~s@^http://(.*?)/get_video\?(.*)video_id=(.*?)@squid://videos.youtube.INTERNAL/ID=$3@){}
elsif
    ($url=~s@^http://(.*?)rapidshare(.*?)/files/(.*?)/(.*?)/(.*?)@squid://files.rapidshare.INTERNAL/$5@){}
elsif
    ($url=~s@^http://(.*?)fbcdn\.net/(.*?)/(.*?)/(.*?\.jpg)@squid://files.facebook.INTERNAL/$4@){}
elsif
    ($url=~s@^http://contenidos2(.*?)/(.*?)@squid://files.contenidos2.INTERNAL/$2@){}
elsif
    ($url=~s@^http://cdn(.*?)/([0-9a-zA-Z_-]*?\.flv)@squid://files.cdn.INTERNAL/$2@){}
elsif
    ($url=~s@^http://web.vxv.com/data/media/(.*?)@squid://files.vxv.INTERNAL/$1@){}
elsif
    ($url=~s@^http://(.*?)megaupload\.com/files/(.*?)/(.*?)@squid://files.megaupload.INTERNAL/$3@){}
elsif
    ($url=~s@^http://(.*?)mediafire\.com/(.*?)/(.*?)@squid://files.megaupload.INTERNAL/$3@){}
elsif
    ($url=~s@^http://(.*?)depositfiles\.com/(.*?)/(.*?)/(.*?)@squid://files.megaupload.INTERNAL/$4@){}
elsif
    ($url=~s@^http://(.*?)\.files\.youporn\.com\/(.*?)\/([0-9a-zA-Z_-]*?\.flv)\?.*@squid://videos.youporn.INTERNAL/$3@){}
elsif
($url=~s@^http://(.*?)\.tube8\.com\/(.*?)\/([0-9a-zA-Z_-]*?\.flv)\?.*@squid://videos.tube8.INTERNAL/$3@){}
elsif
    ($url=~s@^http://(.*?)\.tube8\.com\/(.*?)\/([0-9a-zA-Z_-]*?\.flv)@squid://videos.tube8.INTERNAL/$3@){}
elsif
    ($url=~s@^http://(.*?)megaporn\.com\/files\/(.*?)\/(.*?)@squid://files.megaporn.INTERNAL/$3@){}

print "$url\n"; }

Since this happen for quite some time and I'm busy coding something else, I don't remember which part were changed. Maybe somewhere around url_regex. Perhaps if piju's blog are still running (which is currently not), we can do a diff to compare them. Anyway, here it is. These scripts are shared and distributed as is. If you change it, feel free to let me know. Otherwise, just enjoy!

Python Server Pages Example of GET POST Form Input

Previously, I've talked about mod_python.publisher method of creating a simple website using Python. And this time, I will talk about mod_python.psp where PSP stands for Python Server Pages.

What good about Python Server Pages for former PHP or JSP programmer is the way you can write your code where you can use the <% # your code here %> tag as you always do. But before that, you have to configure your server to enable mod_python.psp.

Pre-requisite
In order to run your site with PSP, you have to install and enable mod_python. Here is the steps if you are using Ubuntu:

1. Install apache2 and libapache2-mod-python and enable mod-python
   

   $ sudo apt-get install apache2
   $ sudo apt-get install libapache2-mod-python
   $ sudo a2enmod python

2. Edit /etc/apache2/sites-enabled/000-default. Search for this lines:
   

   <Directory /var/www/>
                   Options Indexes FollowSymLinks MultiViews
                   AllowOverride None
                   Order allow,deny
                   allow from all
           </Directory>
   

   and add python handler like this:
   

   <Directory /var/www/>
                   Options Indexes FollowSymLinks MultiViews
                   AllowOverride None
                   Order allow,deny
                   allow from all
                   
                   AddHandler mod_python .psp
                   PythonHandler mod_python.psp
                   PythonDebug On
                   
           </Directory>
   

3. Restart your apache
   

   $ sudo /etc/init.d/apache2 restart

4. You can also enable index.php for your DirectoryIndex by editing /etc/apache2/mods-enabled/dir.conf and add this string (in red):
   

   <IfModule mod_dir.c>
             DirectoryIndex index.psp index.html index.cgi index.pl index.php index.xhtml index.htm 
   </IfModule>
   

Sample Python Server Pages Website
And here is my sample code:

<%
import cgi

thetitle = "Coder's Talk Python Form Example"

# it doesn't matter whether it is post or get method
# getfirst will find the item.
username = form.getfirst('username')
themail = form.getfirst('usermail')

%>
<html>
<head>
<title><%= thetitle %>
</title>
</head>
<style>
body {font-family: Arial; font-size:13px;}
#contentbox {width: 640px; margin:0 auto;}
.copy, .copy a {width: 640px; margin:0 auto; color: #DD8888;}
.formresult {background-color:#FFFF99;display:block;width:100%;padding:10px;}
</style>
<body>
<div id="contentbox"><h1><%= thetitle%></h1>
<%
if username:
    username = cgi.escape(username)
%>
<p class="formresult">Hello <%= username %>!<br/>
<%
    if themail:
        themail = cgi.escape(themail)
%>
Your email is <%= themail %></p>
<%
    else:
%>
You didn't enter your email. You don't have email?</p>
<%
    # end if themail
# end if username
%>
<p>This is my example form page processing using POST method</p>
<form name="myform" method="post" action="index.psp">
Name: <input type="text" name="username"/><br/>
Email: <input type="text" name="usermail"/><br/>
<input type="submit" value="Submit"/>
</form>
<br/>

<p>This is the same thing using GET method.</p>
<form name="myform" method="get" action="index.psp">
Name: <input type="text" name="username"/><br/>
Email: <input type="text" name="usermail"/><br/>
<input type="submit" value="Submit"/>
</form>
</div>
<div class="copy">
<iframe src="http://www.facebook.com/plugins/like.php?href=http%3A%2F%2Fwww.facebook.com%2Fpages%2FCoders-Talk%2F138805102822113&amp;layout=standard&amp;show_faces=false&amp;width=640&amp;action=like&amp;font=tahoma&amp;colorscheme=light&amp;height=35" scrolling="no" frameborder="0" style="border:none; overflow:hidden; width:640px; height:35px;" allowTransparency="true"></iframe>
<br/>&copy; M. Fauzilkamil Zainuddin 
| <a href="http://coderstalk.blogspot.com">Coder's Talk</a></div>
</body>
</html>
<%
# end of code. Copyright (C) M. Fauzilkamil Zainuddin (ApOgEE) - http://coderstalk.blogspot.com
%>

You can also download the source code here . Make sure you rename it as index.psp or change the action name on the form tag to be the same as your psp file name. Feel free to try it, modified it and don't forget to 'Like' this blog on Facebook too

Python Code to Get IP Address From Interface Name

From my previous code on How To Print Active Network Interface List in Linux using Python, I got this question.

Question from jimchris:
Nice work, now how do we get ip address from ifacedata?

My Answer:
Here's how... Add this function to your python code:

def getifip(ifn):
    import socket, fcntl, struct
    sck = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    return socket.inet_ntoa(fcntl.ioctl(sck.fileno(),0x8915,struct.pack('256s', ifn[:15]))[20:24])

Then use it in your previous code like this:

print getifip(ifacedata[0])

Update: you can also download the python source code here

Wish you luck and Happy Coding!