My Personal Blog

Just a spot for my random thoughts.

Virtualmin, Apache, and Nginx Reverse Proxy

I wanted to be able to setup a reverse proxy with nginx and apache but continue using Virtualmin GPL to manage my domains. The only reason I wanted a reverse proxy is that several of the domains I host utilize a very large amount of images.  I wanted a efficient way to handle these images since apache uses the same process to handle everything for the domain user and thus its memory footprint continued to grow till the process was killed.  Thus I had processes using 150+ MB of memory just to serve a picture.  

Anyway, Virtualmin does not support this config out of box so I had to do a bit of rigging :-)  I figure I would share how I went about doing this.

A bit of background.  Normally with this setup, you would have nginx accessible by the world and apache only accessible by the local host.  But if you plan on continue allowing Virtualmin to manage your domains, you can't do this.  Unless you want to manually edit your config files each time Virtualmin creates a domain.  So instead, I had to leave both open to the world.  

Installing/configuring nginx

Ok, so assuming that you have a fully functioning Virtualmin and Apache2 servers, let's install nginx.  I use Ubuntu 10.04 LTS under Linode and the nginx server available in it's ppa is a bit old.  So let's get the latest version from nginx's ppa:

sudo add-apt-repository ppa:nginx/stable 
sudo apt-get update
sudo apt-get install nginx

Now create the file /etc/nginx/proxy.conf and add the following to it:

proxy_redirect          off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size 10m;
client_body_buffer_size 128k;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffers 32 4k;

Those are the standard settings.  You may have to adjust the numbers to best fit your site's needs.

Open /etc/nginx/nginx.conf and edit it to match the following (adjust to your server's needs)

user www-data www-data;
worker_processes 2;

error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;

events {
worker_connections 1024;
# multi_accept on;
}

http {
include /etc/nginx/mime.types;

access_log /var/log/nginx/access.log;

sendfile on;
#tcp_nopush on;

#keepalive_timeout 0;
keepalive_timeout 65;
tcp_nodelay on;

gzip on;
gzip_disable "MSIE [1-6]\.(?!.*SV1)";

include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}

I created/edited the default host file to handle any domains that do not have a specific config file:

/etc/nginx/sites-available/default

server {
    listen 12.34.56.78:80 default;
    server_name  _;
    access_log /var/log/nginx/default.access.log;
    error_log /var/log/nginx/default.error.log;

    location / {
        proxy_pass http://12.34.56.78:9091;
        include /etc/nginx/proxy.conf;
   }
}

Now create a virtual host file.  I created one for each of my domains and named them like domain.com.conf so that my Virtualmin script can automatically handle creating them (see below).

So for example, create /var/nginx/sites-available/mydomain.com.conf:

server {
    listen 12.34.56.78:80;
    server_name www.mydomain.com mydomain.com foo.mydomain.com bar.mydomain.com;
    access_log /var/log/virtualmin/mydomain.com_nginx_access_log;
    error_log  /var/log/virtualmin/mydomain.com_nginx_error_log;
    location / {
        proxy_pass http://12.34.56.78:9091;
        include /etc/nginx/proxy.conf;
    }

    location ~* ^.+\.(jpe?g|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mp3)$ {
        expires 30d;
        root /home/mydomainuser/public_html;
    }
}

Enable your site:

sudo ln -s /etc/nginx/sites-available/mydomain.com.conf /etc/nginx/sites-enabled/mydomain.com.conf

Of course you will replace 12.34.56.78 with your server's IP address (if you are using a host with multiple IPs, be sure to use the correct one for this host!).  Also change mydomain.com and add any other server names to the server_name directive.  Now for the proxy_pass, normally this would 127.0.0.1:9091 (9091 being the port we are going to configure apache to run on).  But like I said earlier, I still want Virtualmin to be fully functional without me having to manually edit the apache config files after Virtualmin created them to force apache to listen to 127.0.0.1.  So I am leaving both open to the world and thus apache will listen to the host's IP address.

Test your configs with:

nginx -t -c /etc/nginx/nginx.conf

You should get a success message.  If you get errors fix it before starting nginx.

Start/restart nginx:

sudo /etc/init.d/nginx restart


Configuring Apache
 
We need to now configure Apache to listen to port 9091 and to tell it to use the correct IP address in the logs and in the $_SERVER var for PHP.  

Open /etc/apache2/apache2.conf and change NameVirtualHost 12.34.56.78:80 to 12.34.56.78:9091

Open /etc/apache2/ports.conf and change Listen 80 to Listen 9091

Open all the config files in /etc/apache2/sites-available and change to

Install mod_rpaf so that apache knows the true IP address of the user accessing the site:

sudo apt-get install libapache2-mod-rpaf
sudo a2enmod rpaf 

Open /etc/apache2/mods-available/rpaf.conf and add your IP address(es) to the RPAFproxy_ips directive so that it looks something like this:


RPAFenable On
RPAFsethostname On
RPAFproxy_ips 127.0.0.1 12.34.56.78

 Its important to add your server's IP address(es) to the RPAFproxy_ips directive as if you do not, $_SERVER['REMOTE_ADDR']  will always be your server's IP address which can be bad for scripts that rely on this (like php based firewalls).

Reload apache:

sudo /etc/init.d/apache2 restart


Configuring Virtualmin

Now we have to make some changes to Virtualmin and its config files.

First we need to edit all the existing servers to use the new apache port.  So open each file in /etc/webmin/virtual-server/domains (each file represents a server) and change web_port=80 to web_port=9091.

Now, login to Virtualmin and go to Server Settings -> Server Templates -> click on your default template -> choose Apache website from the template section dropdown -> change Port number for virtual hosts to 9091 then click Save and Next.

Restart webmin:

sudo /etc/init.d/webmin restart

Choose Log file rotation from the template section dropdown, choose Log files below for the Additional files to rotate field.  Add the following to the field's textbox:

/var/log/virtualmin/${DOM}_nginx_access_log
/var/log/virtualmin/${DOM}_nginx_error_log

Note, that you will need to manually add the nginx log files to logrotate for existing domains.  You can use Webmin -> System -> Log File Rotation to do so.


Automating Virtualmin for nginx

Now I didn't want to manually create new virtual host files for nginx each time Virtualmin created a server.  So, I created a little script to do it for me.

First, I created a template file /etc/nginx/sites-available/template.conf which has the following in it:

server {
    listen {SITE_IP}:80;
    server_name www.{DOM} {DOM};
    access_log /var/log/virtualmin/{DOM}_nginx_access_log;
    error_log  /var/log/virtualmin/{DOM}_nginx_error_log;

    location / {
        proxy_pass http://{SITE_IP}:9091;
        include /etc/nginx/proxy.conf;
    }

    location ~* ^.+\.(jpe?g|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mp3)$ {
        expires 30d;
        root /home/{USER}/public_html;
    }
}

Now, I created a script that Virtualmin will run after it creates/deletes/modifies a server.

/usr/local/bin/virtualmin.sh

#!/bin/sh
NGINX_CONF_FILE="/etc/nginx/sites-available/${VIRTUALSERVER_DOM}.conf "

if [ "$VIRTUALSERVER_ACTION" = "CREATE_DOMAIN" ]; then
	if [ "${VIRTUALSERVER_WEB}" = "1" ];
	then
		cp /etc/nginx/sites-available/template.conf $NGINX_CONF_FILE

		perl -pi -e "s#{DOM}#$VIRTUALSERVER_DOM#g" $NGINX_CONF_FILE
		perl -pi -e "s#{SITE_IP}#$VIRTUALSERVER_IP#g" $NGINX_CONF_FILE
		perl -pi -e "s#{HOME}#$VIRTUALSERVER_HOME#g" $NGINX_CONF_FILE
		ln -s $NGINX_CONF_FILE /etc/nginx/sites-enabled/${VIRTUALSERVER_DOM}.conf
		/etc/init.d/nginx reload
	fi


elif [ "$VIRTUALSERVER_ACTION" = "DELETE_DOMAIN" ]; then
        if [ "${VIRTUALSERVER_WEB}" = "1" ];
        then
		    rm /etc/nginx/sites-enabled/${VIRTUALSERVER_DOM}.conf
    		rm /etc/nginx/sites-available/${VIRTUALSERVER_DOM}.conf
	    	rm /var/log/virtualmin/${VIRTUALSERVER_DOM}_nginx_*
            /etc/init.d/nginx reload
    	fi


elif [ "$VIRTUALSERVER_ACTION" = "MODIFY_DOMAIN" ]; then
        if [ "${VIRTUALSERVER_WEB}" = "1" ];
        then
    		if [ ! -f $NGINX_CONF_FILE ]; then
 		        cp /etc/nginx/sites-available/template.conf $NGINX_CONF_FILE
	        	perl -pi -e "s#{DOM}#$VIRTUALSERVER_DOM#g" $NGINX_CONF_FILE
		        perl -pi -e "s#{SITE_IP}#$VIRTUALSERVER_IP#g" $NGINX_CONF_FILE
        		perl -pi -e "s#{HOME}#$VIRTUALSERVER_HOME#g" $NGINX_CONF_FILE
		        ln -s $NGINX_CONF_FILE /etc/nginx/sites-enabled/${VIRTUALSERVER_DOM}.conf
	    	fi
    	fi

	    if [ "$VIRTUALSERVER_DOM" != "$VIRTUALSERVER_OLDSERVER_DOM" ]; then
	        if [ "${VIRTUALSERVER_WEB}" = "1" ];
        	then
			    OLD_NGINX_CONF_FILE=/etc/nginx/sites-available/${VIRTUALSERVER_OLDSERVER_DOM}.conf
			    mv $OLD_NGINX_CONF_FILE $NGINX_CONF_FILE
			    rm /etc/nginx/sites-enabled/${VIRTUALSERVER_OLDSERVER_DOM}.conf
        		perl -pi -e "s#$VIRTUALSERVER_OLDSERVER_DOM#$VIRTUALSERVER_DOM#g" $NGINX_CONF_FILE
	        	perl -pi -e "s#$VIRTUALSERVER_OLDSERVER_IP#$VIRTUALSERVER_IP#g" $NGINX_CONF_FILE
		        perl -pi -e "s#$VIRTUALSERVER_OLDSERVER_HOME#$VIRTUALSERVER_HOME#g" $NGINX_CONF_FILE
			    ln -s /etc/nginx/sites-available/${VIRTUALSERVER_DOM}.conf /etc/nginx/sites-enabled/${VIRTUALSERVER_DOM}.conf
	    	fi
	    fi

        if [ "${VIRTUALSERVER_WEB}" = "1" ];
        then
	        /etc/init.d/nginx reload
	    fi
fi

Now go to Virtualmin -> System Settings -> Virtualmin Configuration -> choose Actions upon server and user creation from the category dropdown and add /usr/local/bin/virtualmin-postaction.sh to "Command to run after making changes to a server."  Now Virtualmin will automatically manage the nginx host file for you.


That should do it!

References:
https://help.ubuntu.com/community/Nginx/ReverseProxy
http://www.dikant.de/2008/07/10/nginx-as-a-reverse-proxy-for-apache/
https://www.virtualmin.com/node/19887
 

"I Hope I'm Ready"

My grandmother, Nettie Shrader, went to be with the Lord last night.  She passed with her loving husband of 53+ years, her three sons, several of her grandchildren and friends around her. Unfortunately I was not which pains me greatly. But as my sister texted me the scene, I was with them in my mind and heart.  Despite not being with her in the end, we did get to enjoy a good time of fellowship with her and my grandfather a couple weeks ago.  More on that in a few.

I miss her terribly and have since we flew back home after that last visit.  I knew it would be the last time I got to see her on this side of heaven.  Its been a long night and day so far, but even in the pain of knowing she is gone, in my heart I rejoice.

You see, Granny was truely a Proverbs 31 woman, even if she may have not known exactly what that meant.  

10 An excellent wife who can find?
She is far more precious than jewels.
11 The heart of her husband trusts in her,
and he will have no lack of gain.
12 She does him good, and not harm,
all the days of her life.

In today's culture, it is getting difficult to find a marriage that lasts for more than a few years.  But their's lasted for almost 54 years and would have been more if the Lord allowed it.  I don't really remember them telling each other "I love you" but it was obvious they did. In a time now where "love" is flung around with pizza, their actions said more than words.  They had been through much together and even faced separation once before when Grandpa nearly died back in the 80's with heart issues.  Granny loved Grandpa very much and he loved her.  It was obvious from the way they would pick on each other, the looks they gave each other, the stories they told together.  It was more than obvious when I saw the look in Grandpa's eyes as he watched his loving wife fade into almost nothing.  Remembering that look breaks my heart more than anything.  But the point is why did he have that look?  Because Granny was an excellent wife who did him good all the days of her life.

13 She seeks wool and flax,
and works with willing hands.
14 She is like the ships of the merchant;
she brings her food from afar.
15 She rises while it is yet night
and provides food for her household
and portions for her maidens.

Granny was one of the hardest workers I've ever known.  I vaguely remember her telling stories of working on farms and factories before I was born which had to be difficult labor.  But, after I was born, she worked as an elementary school cafeteria lady and then as a cafeteria lady for a nursing home.  At the nursing home, while in her 60's, she worked weekday, weekend, and holidays to serve the elderly.  All her working days, she would come home to care for her household doing the usual stuff but also helping to raise the grand kids.  Raising kids of her own and then grand kids, always meant needing a well stocked fridge.  Nobody ever went hungry. At 71 she reluctantly retired, but no rest came to the weary.  She found out she had lung cancer the following year. 

20 She opens her hand to the poor
and reaches out her hands to the needy.

Granny almost never turned away a person in need even if that "need" may have really been a "want," she still gave.  Money never really seemed to matter to her.  And it wasn't just money that she gave.  She gave of her time and love.  I remember in elementary school where she worked as one of the cafeteria ladies, she was known as granny to many of the kids.  She would give me and my friends extra portions of the school cafeteria nastiness they called food to find out later she paid for those extra portions out of her pocket.  She was kind and loving to all the kids, no matter what social ring they were in.  Cool kids or the not-so-cool kids (the group I was in), she loved us all.  She was always there to encourage the ones who were sad and laugh with those who were joyous.  Even later when she worked in the nursing home, people loved to come see her.  She was a beacon of light to the orphaned, the widow and widower, and everyone in between.

26 She opens her mouth with wisdom,
and the teaching of kindness is on her tongue.

I often came to her for wisdom and she never withheld it.  It was her words that directed me in the right direction many of times.  She and Grandpa would take us to church and even though that didn't mean much to me until later in life, in hindsight, it was probably the wisest thing she could have ever done for me.   She would tell me the truth, no matter how hard it may have been.  She gently guided me through my parent's divorce and kindly spoke to me when apparently Santa Clause had told her of my Christmas wish for my parents to reunite.  In school she would say, "worry about school and not girls." And I did, or tried, until I met my beautiful wife after I graduated high school and before I was able to finish college.  But Granny rejoiced with us at our wedding and immediately adopted her as grand daughter.

28 Her children rise up and call her blessed;
her husband also, and he praises her:
29 "Many women have done excellently,
but you surpass them all."

She was indeed blessed.  Not only with three sons and many grand children and great grand children but also with many, many others who called her Granny and friend.  No, she was not perfect, but who of any of us can claim to be.  She was what she needed to be: a faithful and love-bound wife, a loving mother, a caring grandmother, a giving hand and servant to many.  Yes, in Grandpa's eyes and ours as well, she surpassed them all. 

30 Charm is deceitful, and beauty is vain,
but a woman who fears the Lord is to be praised. 

Granny feared the Lord.  I remember one year we had a massive flood that kept us at the elementary school for hours after school had ended.  Roads were washed away and the field in front of her house was half way filled with flowing water.  For those of you who remember, or at least know of the field I talk about, it was indeed a massive flood.  I remember sitting with her at the kitchen table watching that water flow by and noticed that tears were rolling down her face.  I asked, "What's wrong Granny?"  She replied with something along the lines of "God is merciful.  He promised Noah he would never flood the world again.  This flood proves his majesty."  She feared God.

When we visited my grandparents on that last trip a couple weeks ago, I knew I had to ask her.  Time was running out as the cancer had rapidly spread throughout her entire body.  I needed to know for sure.  At that same kitchen table looking out the same window, I asked, "Granny, what do you think about the day when you'll be standing in front of Lord Jesus."  

In her small, weak voice she said, "Well (pause) I hope I'm ready."  

"What will make you ready Granny?" I asked.  

"The Lord Jesus."  

"And what did Jesus do?"

"He died on the cross for my sins."

 "And have you asked Him to forgive you your sins?"

"Yes, yes I have.  Many, many times."

I knew.

Granny I miss you so much.  My tears are mixed with pain and joy.  Pain for losing you knowing I'll never see you again with my physical eyes.  But joy because I, without a shadow of doubt, know that you heard last night the words "Well done good and faithful servant.  Enter into the joy of your master."  I eagerly look forward to the day I'll join you in that joy.

My question I leave for my friends and family is, are you ready as she was ready? 

Upgrading Joomla 1.6.x to Joomla 1.7

For those who couldn't easily find the instructions on how to upgrade Joomla 1.6 to Joomla 1.7 and have an earlier version install than Joomla 1.6.5, here's how ya do it.

First download the patch file for Joomla 1.6.0 to Joomla 1.6.5 from here. Then upload it to your Joomla installation's root and unpack. Then you can follow these instructions to upgrade to 1.7 (basically use Joomlas new one click update feature to upgrade).

This will work for Joomla 1.6.0, 1.6.1, 1.6.2, 1.6.3, and 1.6.4.

Ubuntu and the Epson Workforce 520

So, I had trouble getting my Epson Workforce 520 printer working on Ubuntu. Ubuntu recongized the printer and even allowed me to install it recommending drivers, etc. But whenever I tried to print something, the printer would just spit out blank pages. Natty promised to download the propriatory drivers from Epson's website. Well it does but unfortunately doesn't allow you to choose the new driver to use!!

So, here's how I did it.

First, download the driver from OpenPrinting and install it (download the appropriate 32bit or 64bit deb for your system).

Now, open a root terminal or become root (sudo su).

cd /opt/epson-inkjet-printer-workforce-525/ppds/Epson

gunzip Epson-WorkForce_520_Series-epson-driver.ppd.gz

Now, we are going to install the printer via Cups Web Interface so open a browser and browse to http://localhost:631.

Under the middle section, CUPS for Administrators, click Adding Printers and Classes.

Click the Add Printer button.

After CUPS searches and finds your printer, choose the desired printer and click Continue.

Note: If you are installing the Epson via the network (wifi), there will probably be two printers listed that looks something like:
EPSON42B519 (WorkForce 520) (EPSON WorkForce 520)
EPSON WorkForce 520 (EPSON WorkForce 520).

Choose theEPSON WorkForce 520 (EPSON WorkForce 520). When you click Continue, the connection for the printer should be something like "lpd://192.168.1.25:515/PASSTHRU" (with the IP address of the printer of course). If it is not, go back and choose the other.

Near the bottom, choose to manually select a PPD file beside "Or Provide a PPD File." Browse to /opt/epson-inkjet-printer-workforce-525/ppds/Epson and choose the file,Epson-WorkForce_520_Series-epson-driver.ppd.

Click Add Printer, set your printer defaults, and you're done! The printer should now show up and be ready to use.

AirPrint with Ubuntu 10.10

I just upgraded my iPad to iOS 4.2.1. As most have figured out by now, AirPrint is not available for printers outside of the chosen few HP printers. So, of course I set out to find a way to make it happen with my Samsung multifunction printer. To my delight, Ubuntu comes with all you need pretty much out of the box. All I had to do was create a printer service file for avahi, make a couple changes to CUPS config, then restart cups and BAM, I'm printing from my iPad.

Okay, so here are the details. This of course assumes that you already have an installed and functioning printer within Ubuntu.

First, let's create the service file for avahi.

sudo pico /etc/avahi/services/printer.service

Insert the following into the file and save by holding the control key then hit O




My Printer

_ipp._tcp
_universal._sub._ipp._tcp
631
txtver=1
qtotal=1
rp=printers/My-Printer-CUPS-Name
ty=My Printer
adminurl=http://198.168.1.4:631/printers/My-Printer-CUPS-Name
note=My Printer
priority=0
product=virtual Printer
printer-state=3
printer-type=0x801046
Transparent=T
Binary=T
Fax=F
Color=T
Duplex=T
Staple=F
Copies=T
Collate=F
Punch=F
Bind=F
Sort=F
Scan=F
pdl=application/octet-stream,application/pdf,application/postscript,image/jpeg,image/png,image/urf
URF=W8,SRGB24,CP1,RS600

 

The things to change are in red.My Printer can be anything you want. My-Printer-CUPS-Name has to be the name CUPS has assigned to the printer. You can obtain this by going to System -> Administration -> Printing or browsing to CUPS web admin. In Ubuntu's printer manager, it will be whatever is listed underneath the printer image. The IP address in the adminurl should be the IP of the computer that has the printer installed.

Of course you can change the options such as Duplex, etc based on your printers features.

Now, you need to change a couple things with CUPS config.

sudo pico /etc/cups/cupsd.conf

Make sure you have the following:

ServerAlias *
Port 631
Listen /var/run/cups/cups.sock

 

Now go back to System -> Administration -> Printing and click Server -> Settings. Make sure "Publish shared printers connected to this system" is checked. Click OK. Restart the CUPS server.

sudo /etc/init.d/cups restart

You should now be able to print from the iPad!

Ubuntu 10.04, suPHP, and phpMyAdmin

I've recently setup an Ubuntu server that uses suPHP. I wanted to also use phpMyAdmin but found that there were some issues with getting it to work with phpMyAdmin. I found some work arounds while Googleing but nothing definitive. So this is how I did it.

The first problem I ran across was that suPHP would not allow phpMyAdmin to run because Ubuntu installs it into /usr/share/phpmyadmin rather than the default web document root /var/www. So suPHP would return the error " File "/usr/share/phpmyadmin/index.php" is not in document root of Vhost "/var/www." Some got around this by simply installing phpMyAdmin manually in /var/www but I'd rather let Ubuntu handle it. (suPHP would not allow symlinks either).

The easiest way to get around this would be to tell suPHP to ignore that security restriction by editing /etc/suphp/suphp.conf and change

;Check wheter script is within DOCUMENT_ROOT
check_vhost_docroot=true

to

;Check wheter script is within DOCUMENT_ROOT
check_vhost_docroot=false

But that screamed "SECURITY BREACH" to me so that was not an option.

So, what I did was make phpMyAdmin its own virtual host.

Edit Apache's phpMyAdmin config file:

sudo pico /etc/apache2/conf.d/phpmyadmin.conf

At the beginning of this file add:

ServerAdmin webmaster@localhost
DocumentRoot /usr/share/phpmyadmin

At the end of the file, add

Save by holding down the Control key and hit the o key. Then ctrl-x to exit.

Then, tell suPHP that /usr/share/phpmyadmin is okay:

sudo pico /etc/apache2/mods-available/suphp.conf

Add the following before "":

suPHP_Engine on

And in /etc/suphp/suphp.conf, add phpmyadmin's directory to an allowed docroot:

;Path all scripts have to be in

docroot=/var/www:${HOME}/public_html:/usr/share/phpmyadmin

Now, after browsing to http://mysite.com/phpmyadmin, I got a 500 Internal Error message. Searching the logs, I found this message:

UID of script "/usr/share/phpmyadmin/index.php" is smaller than min_uid

Two issues. One is that the Ubuntu's phpMyAdmin install is owned by root. Easy fix:

sudo chown -R www-data:www-data /usr/share/phpmyadmin

But, still get the same error. So the second reason is that suPHP is configured to have a minimum UID of 100. www-data's UID is 33. So, edit suPHP's config to change the minimum to 33.

sudo pico /etc/suphp/suphp.conf

Change:

; Minimum UID
min_uid=100

; Minimum GID
min_gid=100

to:

; Minimum UID
min_uid=33

; Minimum GID
min_gid=33

Make sure you change the min_gid as well or else suPHP will still reject phpMyAdmin. I guess this could be a potential security issue as well but less of a one than letting web scripts have free rein outside their designated directories.

Now we have suPHP and phpMyAdmin working together!

The birth of JHarvest - Harvest integration into Joomla

Since I've started to take on clients, I've needed a decent way to track my time and invoices without much overhead. Way back, I purchased a Joomla extension called JForce to handle this but found that it was inadequate for my needs for time tracking. Recently, I came across Harvest. Harvest is an online service that tracks time and can even manage estimates and invoices. It's time tracking interface was nice and came with an iPhone app that allowed me to easily track my time. But unless I paid a monthly fee, I was limited to the number of projects and clients I could create. So, I created one Freelancing project with a task for each of my clients. Using Harvest's reporting features, I was able to differentiate between my clients for time entries.

I couldn't use Harvest's invoicing feature because of the method I was using it. Plus, I wanted to be able to have more control over my invoices, who had access to them, etc. So I used JForce for invoicing and Harvest for time tracking. JForce allowed me to create invoices via my own site and give clients access to only their own invoices. Some of my clients asked if there was a way for them to see how many hours of work I have done for them at any given time. I didn't have a way to do this of course since Harvest didn't have a way to display time sheets to clients. Thus, the thought of JHarvest was born.

At first I thought that I would simply create a Joomla extension that pulls information from Harvest and displays it to the user. This is still my plan so that JHarvest can be a standalone extension. It will allow admins to match Joomla users with JHarvest clients. Any information for that client will be made available for viewing to the matched Joomla user via Joomla's frontend. I have yet to fully design how this will function and look so feedback on this would be nice!

But in addition to that, I plan to hook it into other components. JForce just rebranded themselves as JForce PM. This time, I think they got it right for the most part. The only thing really lacking is that they do not have the time tracking features of Harvest such as timers and decent time sheet reports. So in addition to JHarvest being a standalone extension, I'm creating it to have a "plugin" system where it can be extended to sync with third party extensions. The first will be JForce where basically the user will be able to sync Harvest information into JForce. I'm creating three sync modes for this. The first is "Project" which will sync project to project, task to task, time entry to time entry, etc. The second will be "Task" where the admin will be able to define a default JForce task (which is by nature assigned to a specific project) for each Harvest project assigned task. Each Harvest time entry will then be synced as a JForce time entry for the defined JForce task. The final mode will be "Time Entry" where the admin will be able to assign each Harvest project assigned task to a JForce project. Each Harvest time entry will either a) become a new JForce task with the entry's note as the task's name (and of course a new JForce time entry as well to track hours). Or, just a new JForce time entry if the entry does not have any notes and it will be associated with the defined default JForce project/task.

So far, I have the backend almost fully functional. It downloads all Harvest data and stores it locally. That data is kept up to date with each sync. I have it 90% interfaced with JForce (enough to where I can use it for my purposes). There are a few things left to do with the backend before I start working on the frontend. First, it needs a safe way to delete local data that no longer exists in Harvest if the admin chooses to. The other 10% of JForce integration also needs to be completed. Finally, I need to design/code ways to display properties such as cost per hour, budget, etc for individual projects, tasks, and contacts. Once that is done, I'll start coding the frontend.

Eventually I would like to be able to feed data back into Harvest. But that will be after I have the initial release of JHarvest available. My first goal is to simply be able to retrieve and present Harvest data to Joomla users in a meaningful way. Then, I'll start adding new features such as feeding data back to Harvest and support for multi-user Harvest accounts.

I would love to hear about ways this would be useful to the Joomla community, if at all :-). If nothing else, I'll use it extensively until JForce designs an iPhone app and improves their time tracking features ;-). Would you be willing to pay for such an extension? If so, how much? In what ways would you like it to present data to the Joomla end user?

Leave comments below.

How to set up a VNC web (Java) client for Ubuntu

I've had to play with this every time I reinstall Ubuntu and want to setup a VNC web client which is really handy when I'm work and need to access my PC at home for whatever reason.

First, make sure you have remote desktop enabled by going to Menu -> System -> Preferences -> Remote Desktop. Click "Allow other users to view your desktop" and "Allow other users to control your desktop." For my use, since I VNC into my desktop from remote locations and thus will not be at my desk to accept my own connection, I uncheck "You must confirm each access to this machine" and check "Require the user to enter this password." Make sure you use a strong password. Of course, you may need to configure your router to forward incoming communication for ports 5800 and 5900 to your desktop you're wanting to connect into.

Second, if you do not already have apache installed, install it:

sudo apt-get install apache2

Third, download the tightvnc-java package. I use the latest version at Tightvnc's website rather than the one in Ubuntu's repositories. Download and extract the java Binary *.class and JAR files in Zip archive (or Tar+Gzip) and extract the contents wherever you want.

Finally, open Nautilus as root by hitting Ctrl-F2 and typing gksu nautilus (or if you're a Linux pro, you can use a terminal to copy the contents to /var/www/vnc). Browse to /var/www and create a folder named vnc. Copy the contents of the java vnc viewer to /var/www/vnc. Open index.html and replace all the contents with this:


<br />TightVNC desktop<br />
WIDTH="800" HEIGHT="632">





TightVNC site

And that should do it. You can now access it via http://ip.address/vnc or use a dynamic DNS service such s DnsExit to create a domain name that points to your home IP.

Cron job script to purge Joomla's expired cache

I found that Joomla's caching feature can cause my server to run out of space quickly unless the admin consistently purges expired cache. Since Joomla only has a manual mechanism in place to be able to do this, I decided to create a script that can be run by a cron job to handle this for me.

Its quite simple. First, I load Joomla's framework then simply call the function to purge the cache.

$mainframe = startJoomla();

jimport('joomla.filesystem.folder');
jimport('joomla.filesystem.file');

$cache =& JFactory::getCache('');
$result = $cache->gc();

if ($result) {
echo 'Expired cache purged.';
} else {
echo 'Failed to purge expired cache.';
}

function startJoomla()
{
define('_JEXEC', true);
define( 'DS', DIRECTORY_SEPARATOR );
define('JPATH_BASE', dirname(__FILE__) );
// load joomla libraries
require_once JPATH_BASE . DS . 'includes' . DS . 'defines.php';
require_once JPATH_LIBRARIES . DS . 'loader.php';
jimport('joomla.base.object');
jimport('joomla.factory');
jimport('joomla.filter.filterinput');
jimport('joomla.error.error');
jimport('joomla.event.dispatcher');
jimport('joomla.event.plugin');
jimport('joomla.plugin.helper');
jimport('joomla.utilities.arrayhelper');
jimport('joomla.environment.uri');
jimport('joomla.environment.request');
jimport('joomla.user.user');
// JText cannot be loaded with jimport since it's not in a file called text.php but in methods
JLoader::register('JText', JPATH_BASE . DS . 'libraries' . DS . 'joomla' . DS . 'methods.php');
JLoader::register('JRoute', JPATH_BASE . DS . 'libraries' . DS . 'joomla' . DS . 'methods.php');

$mainframe = & JFactory::getApplication('site');
$GLOBALS['mainframe'] = & $mainframe;
return $mainframe;
}

I put this code in a file in Joomla's root directory. Then I created a cronjob to have this script ran by PHP every night. My cronjob looks something like this:

/usr/bin/php /home/username/public_html/purgecache.php

And that's it! Automatic purging.

ATI driver issues in Ubuntu 10.04 Lucid

Well, it never fails that when I upgrade Ubuntu, I run into a few issues. Gotta love open source. :-)

The issue was with my ATI drivers. I had initially installed them using ATI's script for the catalyst 10.4 driver which is suppose to have support for Ubuntu 10.04. But after installing it, every little movement was super choppy and jerky. So I decided to remove it. Usually easy enough right? Just a sudo apt-get remove fglrx and success? Nope.

Removing fglrx ...
dpkg-divert: mismatch on package
when removing `diversion of /usr/lib/libGL.so.1.2 to /usr/lib/fglrx/libGL.so.1.2.xlibmesa by fglrx'
found `diversion of /usr/lib/libGL.so.1.2 to /usr/lib/fglrx/libGL.so.1.2.xlibmesa by xorg-driver-fglrx'
dpkg: error processing fglrx (--purge):
subprocess installed post-removal script returned error exit status 2
Processing triggers for ureadahead ...
Errors were encountered while processing:
fglrx

It failed. After searching for a long time, I finally came across a discreet page that gave me what I needed to make it happen.

sudo dpkg-divert --rename --remove /usr/lib/libGL.so.1.2
sudo dpkg-divert --rename --remove /usr/lib32/libGL.so.1.2 sudo dpkg --force-all --purge fglrx

Success! Afterward, just make sure you reinstall either xorg's open source ATI driver or Ubuntu's fglrx package as you might find yourself in low graphics mode otherwise :-) You may also need to do

sudo dpkg-reconfigure xserver-xorg

Later...I found this post which helped tremendously as well.

Western Digital My Passport SE 1TB Portable Drive

So, my well used WD 160GB portable drive started clicking. I heavily use it to store my web development files and when the clicks started, I became very nervous. So, I did a search for a new drive. Well, as I often do, I impulsively bought the WD My Password SE 1TB drive from Amazon thinking, wow that much space, so portable and decently priced! (It was as much as my 160GB I bought two years ago). I began nightly backups (wonderful tool rsync is) until my new drive arrived.

It finally arrived yesterday. I was so impressed with my slick, little drive with so much data capacity. I quickly reformatted (to ext3 of course) and synced my files from the old to the new. Then I had the wake up call. I picked up the drive to move some cables around and bam, connection was lost and bye bye all unsaved data. What the crap? Turns out, the slightest little touch of the cable causes it to lose connection, and I do mean the slightest touch. Wouldn't mind it so much if this wasn't a portable drive. Who designed this thing? They should be fired.

2009 Newsletter

Dear Family and Friends,

A belated Merry Christmas and a Happy New Year to you all! We hope that 2009 treated you well and that 2010 will be full of new mercies and blessings.

It’s hard to believe that we have begun our third year in Houston, Texas. We are far from becoming true “Texans” though. Thus far, we have resisted buying over-sized vehicles, using phrases such as “fixin to,” calling all non-alcoholic carbonated beverages coke or buying a Republic of Texas flag. Despite Texas' attempt to indoctrinate Alan with Texas government and history this past spring, we pretty much remain Virginians with one exception. We've become yellow-bellied when it comes to the weather as we are in jackets when its only seventy degrees! We can't complain though as seventy degree winters are quite nice.

We celebrated our fifth anniversary this past week which is another hard to believe. At this rate, we'll be celebrating our 50th before we know it! There has been some bumps along the way, but each year has truly been more fulfilling than the previous. We both eagerly look forward to what the Lord has planned for our lives as we grow old together.

Pancho and Felíz are doing quite well. They are as crack-headed as ever but have been fun and good little companions. We couldn't imagine life without them.

Last year at this time, things at work were uncertain due to the aftermath of Hurricane Ike. Alan's department was closed and thus he was transferred into a new department created post Ike. What was directly caring for patients turned into an eight hour day desk job. After several months of working behind a desk while locked behind bars, Alan had just about enough. He finally started to apply to local hospitals closer to home. By the grace of God, he busted out of prison in March when he accepted a job at a local hospital. He is now working on a medical/surgical/telemetry floor and loving it. Not only is it just ten minutes down the road but also a much better working environment with a lot of opportunity. He has already been awarded two bronze pins for patient recognition and is the co-chair for the hospital's Clinical Informatics Committee.

Eda still loves her job. It has been a bit of a change in pace this year as the school district decided to change from a block schedule to a seven period schedule. This year she is teaching several pre-advanced placement courses in Spanish and has been recognized for her talent by being a finalist out of hundreds of teachers for the HEB Teacher Recognition Award. The highlight of her year has been her role as the advisor to Clear Brook's CRY (Children Rights and You) chapter. This is a national organization that fights for the rights of children. Clear Brook's chapter has been involved in several community services including adopting local families to provide Christmas gifts and volunteering at the Boys and Girls Harbor.

Speaking of the Boys and Girls Harbor; we have had a couple opportunities to serve through our church and through CRY. It is a temporary home for 4-18 year old children whose parents are not able to care for them for whatever reason. These little guys are full of energy, smiles, and hope despite their dire circumstances. We wanted so much to pack them all up and bring them home with us. It is heart breaking knowing that most of them do not have a home to go to for Christmas. If you think of it, please pray for these little guys. We hope to become more involved with this ministry over this next year.

We pray that God blesses you greatly in 2010 and that He shines His Light upon you. We love and miss you all!

With Love,


Alan, Eda, Pancho and Felíz.

Tasktop - A must have Eclipse plugin

I'm always on the search for a decent piece of software to help track the time I spend on various coding projects for clients. Before I found Tasktop, I was already using Mylyn tasks to plan out items I needed to do for various projects. But I used a timer no my phone, then manually recorded the time into another project tracker such as Zoho Projects, JForce, or Trac. A pain in the you know what if you know what I mean. As I usually do when I have had enough with my current method of doing things, I do a google search to see if anything new presents itself. Well, this time my search led me to Tasktop and I was sold as soon as I tried the trial.

Tasktop is not the perfect solution for me. The perfect solution is an all-in-one project tracker, invoice and quote manager. The project trackers above fall short and/or are expensive for small-scale freelancers like myself. However, Tasktop has solved my time tracking issues and has made it very convenient to code for clients within Eclipse and track billable hours. And, if I use a supported connector, Tasktop will keep my tasks and time in sync with various project trackers!

I'm not going to layout all of Tasktop's feature as you can head over to Tasktop.com to see and experience it for yourself. However I will layout a couple of my favourite features.

1) I love the feature that opens all the windows you were working on when you deactivated a task. This makes sure that only windows (editor or built-in browser) that are pertinent to the task are open. It is such a convenience and a time saver!

2) I live off Google calendar and Tasktop allows syncing tasks as calendar events to my calendar! It will even display the current events in the bottom left hand corner of Eclipse. It gives you the option to sync the due date/time and or scheduled date for the task.

3) Of course my most favourite and the reason I found Tasktop to begin with, is the time tracker capability. I can add/edit/remove time items to be billed. The ONLY thing I do not care for and have communicated to Tasktop is the way the "No Task Active Time." Basically it tracks the time that Eclipse is open but a task is not activated. However, I'm bouncing back and forth between items for clients and I have not found a way to tell it to divvy out no task active time between the clients appropriately. So, I'm forced to just ignore it. Myabe I'm configuring it wrong, I dunno. I hope I am and will be enlightened soon ;-)

Anyway, Tasktop pro is $99 bucks but for me, it is well worth it.

New Google Apps Mail Look

I like it. Looks much more professional and not so "bright and bubbly." I like the label drop down where I can select as many labels as I want by selecting or deselecting the check boxes then clicking Apply. Rows look more tight as well. The colors are more tame and appealing.

I was looking forward to themes but I like the new look so much, now I really don't care.

Do you like the new look? Why or why not?

2008 Newsletter

Dear friends and family,

We hope this letter finds you well. Last year when I sat down to write this letter, it was on our back porch under sunny skies and 75 degree weather. Today, I am bundled up in my warmest fleece because its 34 degrees - For Houstonians, that’s cold! Don’t fret though, they are predicting 72 degrees for Christmas day and we’re OK with that.

I guess we are officially Houstonians since we have been here just over a year now. Pops helped to start off 2008 with excitement for us. I got a call from him in the first half of January – he was in the hospital. He had driven himself there the night before after suffering through five days of abdominal pains. Turns out it was appendicitis. They removed it just before his 80th birthday. Within six weeks he was back to his old self working full time in his shop and as spunky as usual. We are thankful for God’s healing hand in his life.

For Alan’s birthday in February I decided to kidnap him and take him to Austin to check out the music scene. I guess I shouldn’t quit my day job since it was brutally cold and we just couldn’t get into the bar hopping, music scene. Instead we both got nasty colds and spent the weekend running to wal-mart to buy cold medicine. Happy birthday Alan…

For Spring break we were blessed to have Nana, Sally and Ashley come to spend the week with us. We took in the Houston Livestock Show and Rodeo. That was a great cultural adventure for all – Thank goodness Nana and Sally yanked me away from the pooping cow just in time!

The school year ended well and before we knew it our summer travels had begun. We met up with our dear friends the Hamptons (the elder and younger ones) in San Antonio. It was great to spend time with all of them and to take in the sights. Shortly there after we embarked on a two week road trip with the dogs up to Virginia. Along the way we spent some time in Savannah, GA sightseeing, Wilmington, NC visiting Alan’s little brother Ethan and in Monroe, NC with our friends the Quinns. The time in Virginia went quickly, but we are thankful for all the friends and family we spent time with.

The big news maker for us has been hurricane Ike. The school year had barely begun and we were out again for what we thought would be a couple of days for this “minor, category 2” storm. We took my parents and the dogs and evacuated to San Antonio for a couple of days just to be safe. Thankfully my parent’s home was spared and they had power as soon as we returned. We lost a tree and power for about a week, but the weather was cool and it was a good time to reflect on God’s goodness and provision. Many others were not so fortunate. We were out of school for two weeks because there was no power, 39 of our 40 schools were damaged, many students’ homes were uninhabitable and there was simply too much debris for the school buses to move around safely. The hospital where Alan works was severely damaged. It took in about four feet of water. We were very fortunate that he was paid for the time that he could not return to work, and that he still has a job. Nearly 4,000 employees of that hospital system have lost their jobs. Again, we see God’s graceful provision in our lives. Things will not return to normal at that hospital and Alan plans to start the job search in January. All in all it’s been an adventurous and blessed year. We even got some SNOW here in Houston a couple of weeks ago. The kids were thrilled to make snow men (even though it required scooping up all the snow on the block). It was a great way to lift spirits here during the Christmas season. Most of all we are grateful for God’s gift to us in His Son Jesus Christ this Christmas season. For despite the circumstances of the word around us and the burden of our sins, God gave us redemption and hope by faith in His Son, Jesus, born to us 2,000 years ago. We pray that you would be able to fully enjoy this most precious gift during the Christmas season.

With love,

Alan, Eda, Pancho and Felíz

I've been invited to join the jFusion team!

I recently came across jFusion, a powerful and revolutionary Joomla component to bridge Joomla with various softwares. I started testing out its plugin for vBulletin (the forum software used on this site). After posting some questions, hacks, and suggestions, I offered to help out in developing the vBulletin plugin. I was soon afterward invited to join the team! I, along with another gentleman named Haythem, will be maintaining the vBulletin plugin.

We have already made huge leaps in getting the vBulletin plugin working well with Joomla! The next release of jFusion (set for December 25th) will include the newly enhanced vBulletin plugin! But if you can't wait, check out our SVN at http://code.google.com/p/jfusion.

Getting 32bit Eclipse with Aptana to work on Ubuntu Ibex 64bit

UPDATE 2010 May 23: For Ubuntu 10.04 (Lucid), it is no longer required to download and install the individual 32bit libraries. Just follow step number 2 and 9-12.

I decided to upgrade my version of Eclipse to the latest, 3.4. I have a 32bit laptop and a 64 bit desktop. I hate having to maintain two copies of Eclipse so I maintain one 32bit version of Eclipse on my jump drive. Obviously, this poses some problems with my 64bit desktop. There a ways to get a 32bit version of Eclipse fully functional on a 64bit OS. With 3.3, I had no problems doing this. But with 3.4, I ran into some new issues. Specifically with getting the Aptana plugin to work properly. Nothing in the aptana perspective would display and it would crash with errors relating to it not finding a suitable xulrunner.

This is how I ended up doing it. I used parts of the how to found here on the Ubuntu forums posted by linuxed.

1) I downloaded the following packages:

Eclipse
libnss3-1d 32-bit
xulrunner-1.8 32-bit
libnspr4-0d 32-bit
libstartup-notification0 32-bit
libhunspell-1.1-0 32-bit

2) Install the following via apt

sudo apt-get install ia32-libs ia32-sun-java6-bin

3) Create a temporary folder lib32

mkdir lib32

4) Extract the above packages, then extractdata.tar.gz in each of the library packages.

5) Copy the following files from the corresponding packages' data.tar.gz to the lib32 folder you created

libnss3-1d: usr/lib:

  • libnss3.so.1d
  • libnssutil3.so.1d
  • libsmime3.so.1d
  • libssl3.so.1d

libnss3-1d: usr/lib/nss:

  • libfreebl3.so
  • libnssckbi.so
  • libnssdbm3.so
  • libsoftokn3.so

Copy all the files in usr/lib for libnspr4-0d, libstartup-notificaton0 and libhunspell-1.1-0

6) Place the entire xulrunner folder in the lib32 folder

7) Place the entire eclipse directory wherever you want to run it from. For me, it was my jump drive. For you it may be /opt, /usr/lib32, or whatever.

8) Copy the entire lib32 directory to /usr/lib32

sudo cp -R lib32/* /usr/lib32/

9) Create the following file:

sudo gedit /usr/bin/eclipse

10) Place the following into the file and save it

export PATH=/direct/path/to/eclipse/installation/:/usr/lib/jvm/ia32-java-6-sun/bin:/usr/lib32/:$PATH
export JAVA_HOME=/usr/lib/jvm/ia32-java-6-sun/
export MOZILLA_FIVE_HOME=/usr/lib32/xulrunner/ cd /direct/path/to/eclipse/installation/
./eclipse $*

11) Make it executable

sudo chmod +x /usr/bin/eclipse

12) Run eclipse from either the application launcher or a terminal

To get Aptana installed and working, do the following

1) Open Eclipse and intall the SDK. Click Help->Software Updates. click on the Available Software tab and type in"eclipse sdk." Click on "Eclipse SDK" and then install.

2) Once installed, restart eclipse. Then go to Window -> Preferences -> General -> Capabilities and check "Classic Update"

3) Go to Help -> Software Updates -> Find and Install. Select " Search for new features to install" and click next. Click "New Remote Site" and add "http://update.aptana.com/update/studio/3.2/site.xml"

4) Click Ok then Finish. Install Aptana.

5) Restart and you should be good to go!

Comment below if you have any questions or run into roadblocks.

4000 jobs about to be rifted...I'm thankful

So I work for UTMB under Correctional Managed Care (CMC). UTMB is about to layoff 4000 jobs due to the devastation caused by Hurricane Ike. CMC has promised that our jobs are safe. Because, our hospital was attached to UTMB on Galveston Island and the political battles going on, we will not be able to get back to the way things were for who knows how long, if ever. Thus, we are getting bounced around. Our job duties have changed multiple times thus far. Come Monday, they will change again.

For many, this change comes as huge inconvenience. Many will be forced to take huge pay cuts for they will go from nights (with shift diff) to days. For, many it will be a lifestyle change because we will be going from a matrix (working a few days on then having a few days off) and some from nights to a Monday through Friday 8 - 5 job. For many, it will be difficult because of family obligations that were met under the old schedule but will not under the new. It will be difficult because many are in school and were used to having certain weekdays off to study, go to class, complete assignments, etc. But now, all will change.

However, we still have a job when so many come this month will not. And that alone should overshadow all the inconviences stated above. Yet, so many people cannot for whatever reason see that and continue to wallow in their sorrows and inconveniences. I don't understand it.

We have all been promised that once things are back in order, we will be reuninted as a team, better yet, a family. But it will take time, maybe even months. We must learn to count our blessings, focus on the positives, stick it out and move on. We must keep in mind that "this too shall pass."

My excellent wife

An excellent wife who can find? She is far more precious than jewels. ~Proverbs 31:10

My wife is so excellent. She is so patient, loving , caring, supporting, and sacrificial. I am so undeserving of such an excellent wife! But now that I have her, I would not trade her for all the jewels of the world, the world itself, or even the universe!

Assembla.com

So I stumbled upon assembla.com back in the early stages of phpScheduleEm and instantly fell in love.  At the time I was hosting my own SVN/Trac server but was a pain because of issues with my ISP.  I did a search to see if there were any services out there to host svn/trac for me.  During that search, I happened to stumble upon assembla, a feature rich site full of tools for the software developer.

At the time, assembla was completely free.  But just within the last week they are beginning to roll out a subscription based service and making all future free acounts public.  They are charging $2 per user and $3 per gigabyte of space used.  I immediately bought a $100 dollars worth of subscription credits.  The price was absolutely unbeatable for the services I get.

It is at assembla.com where I keep track of all my tasks, bugs, and features and host the latest code.  Users do not report bugs here, but I put the bugs into the trac system once they are confirmed for this is strictly my development workspace.  (Bugs should be reported using this site.)  I use eclipse to develop my applications which has a mylyn trac connector which allows me to easily manage my tickets as I'm developing.  Eclipse also allows me to manage my svn repository.

Anyway, I digress.  I highly recommend assembla.  It not only provides svn and trac but a slew of other services such as Mercurial, Git, time keeping, wiki, milestones, tickets, files, chat, and other to collaborate between users developing the same project.  The space interface is customizable giving it the look and feel of however you want.  I have not had any problems with it thus far except for the minor detail of not being able to edit a comment of a svn commit.  I've contacted support and will hopefully get this worked out soon.

Check them out!