The reason I have not posted anything over the last couple of weeks is because I've been very busy working on the migration of our applications from Ingres 2.6 to the latest 2006 R2 release. We had originally intended to ship this out in late July, but we are now looking (and hoping!) that things will be ready to ship by the end of next week.
So why are we late?
As I've reflected on it, I'm reminded that most IT projects fall into one of the following categories:
1) Release it only when it's ready (typical for many open source projects)
2) Release on time and then apply numerous patches later
3) Release late (and probably over budget)
Vista is probably the most famous project that was late, and this week Microsoft announced that Server 2008 will be delayed. Tonight I read that KDE 4.0 will be delayed by two months, and almost every government IT project is both late and overbudget.
In our case, the problem has been the discovery of a number of bugs that have required patches (for the database) or changes to our code. These have been difficult to predict and requires regression testing (the fact we want to simultaneously release on four platforms only adds to the delay). Now this is not intended to be a cheap excuse, but it does highlight a problem that affects all parts of the industry.
As long as projects like this are "stabs in the dark" (caused by using new and sometimes unfamiliar technology), I'm not sure how things can improve.
Friday, 31 August 2007
Wednesday, 15 August 2007
Managing Brightstor ARCserve Backup
At work, we use CA's Brightstor ARCserve Backup (BAB) to backup our network of Solaris, Linux, Windows and AIX servers. The backup is done on a Solaris 10 server running version 11.5 of the server software.
We connect to a 40 tape Plasmon library equipped with two LTO2 drives (although we only use 1 tape drive at the moment and allocate only 20 slots to the backup server). This connects to our SAN using an ATTO FibreBridge 2300 (SCSI in one end, Fibre Channel out the other).
BAB is fairly good at what it does, but the FibreBridge is a bit dodgy and occasionally crashes. Something it did last Friday night.
Like many organisations, we perform a full system backup on a Friday night, and then use differentials (level 1) backups Monday through to Thursday. Monthly tapes are then stored off site.
One issue I've been fighting with is that BAB stores it's backup catalogs in an Ingres database. Each file that is backed up gets a row. We've been running the backup for about 4 months and now have 61 millions rows in one of the tables.
In order to reduce the size of the database, BAB includes a "prune" option which removes the details of old jobs from the database. I configured this to run, but then noticed that a number of the text catalogs from previous jobs had not been imported into the database (BAB writes these during the backup and then loads them into the database after each session using the MergeCat utility).
So most of yesterday was spent completing the MergeCat (check /opt/CA/BrightstorARCserve/dbase/tmpcat to see what needs importing) and then running the backup last night. I have now just kicked off the dbclean to purge the jobs.
To make things a bit faster, I did the following:
By default, the Ingres configuration only had one write behind thread. This causes a bottleneck as the log buffers filled up before they were being written out. I've increased this to 20.
The dbclean does a bunch of "delete from astpdat where sesid = x" transactions and if the table is big, the transaction log file will fill. I'm currently running a 5GB transaction log file.
The logging system was still choking. Running logstat -statistics showed that I was getting a lot of log waits and log split waits. I increased the buffer count to 200 (from 20) and changed the block size from 4k to 8k (reformatting the transaction log file in the process).
The biggest performance gains came from dropping the indexes. There are three created by default, so every insert or delete also has to update the indexes. In order to speed things up, I dropped all indexes during the MergeCat, and recreated only ix_astpdat_1 for the dbclean (ix_astpdat_1 is keyed on the sesid which is used during the delete - without it would be a nightmare as astpdat is a heap table).
This is causing things to motor along now, although I still want to do some cache analysis to see whether increasing the DMF cache will improve things.
Once the dbclean completes, the only thing remaining is to modify the database (and hopefully reclaim a load of space as heap tables don't compact when rows are deleted) and then recreate the indexes.
We connect to a 40 tape Plasmon library equipped with two LTO2 drives (although we only use 1 tape drive at the moment and allocate only 20 slots to the backup server). This connects to our SAN using an ATTO FibreBridge 2300 (SCSI in one end, Fibre Channel out the other).
BAB is fairly good at what it does, but the FibreBridge is a bit dodgy and occasionally crashes. Something it did last Friday night.
Like many organisations, we perform a full system backup on a Friday night, and then use differentials (level 1) backups Monday through to Thursday. Monthly tapes are then stored off site.
One issue I've been fighting with is that BAB stores it's backup catalogs in an Ingres database. Each file that is backed up gets a row. We've been running the backup for about 4 months and now have 61 millions rows in one of the tables.
In order to reduce the size of the database, BAB includes a "prune" option which removes the details of old jobs from the database. I configured this to run, but then noticed that a number of the text catalogs from previous jobs had not been imported into the database (BAB writes these during the backup and then loads them into the database after each session using the MergeCat utility).
So most of yesterday was spent completing the MergeCat (check /opt/CA/BrightstorARCserve/dbase/tmpcat to see what needs importing) and then running the backup last night. I have now just kicked off the dbclean to purge the jobs.
To make things a bit faster, I did the following:
By default, the Ingres configuration only had one write behind thread. This causes a bottleneck as the log buffers filled up before they were being written out. I've increased this to 20.
The dbclean does a bunch of "delete from astpdat where sesid = x" transactions and if the table is big, the transaction log file will fill. I'm currently running a 5GB transaction log file.
The logging system was still choking. Running logstat -statistics showed that I was getting a lot of log waits and log split waits. I increased the buffer count to 200 (from 20) and changed the block size from 4k to 8k (reformatting the transaction log file in the process).
The biggest performance gains came from dropping the indexes. There are three created by default, so every insert or delete also has to update the indexes. In order to speed things up, I dropped all indexes during the MergeCat, and recreated only ix_astpdat_1 for the dbclean (ix_astpdat_1 is keyed on the sesid which is used during the delete - without it would be a nightmare as astpdat is a heap table).
This is causing things to motor along now, although I still want to do some cache analysis to see whether increasing the DMF cache will improve things.
Once the dbclean completes, the only thing remaining is to modify the database (and hopefully reclaim a load of space as heap tables don't compact when rows are deleted) and then recreate the indexes.
Sunday, 12 August 2007
Plans for more upgrades...
Recently, the rumour sites were full of speculation that Apple were killing off the Mac Mini. This little machine offers PC users the ability to experience Mac OS X without needing to buy (and find space for) a new monitor, keyboard and mouse, and it's been on my radar for a while now.
Therefore, the news last week that Apple had upgraded the specification of the Mini was quite exciting, if only to give a bit more confidence that the machine would be around for a while longer.
Following some careful negotiations with T, I will be buying a new Mini when Leopard is released. There is no way that I would buy one now only to have to upgrade the OS in about two months! The addition of the latest iLife 08 will be good, and I'll probably pay out for iWork 08 as well. The latter being very interesting as the new Numbers application provides a new take on the spreadsheet.
There is some uncertainty over whether my Belkin Omnicube will work with it as this has PS/2 ports, and I'm not sure if a USB to PS/2 converter will do the job, so I might need to factor in a KVM upgrade as well.
I'm guessing I'll need to get an external hard disk for the Mini to get the benefit of Time Machine, perhaps something that will match the style of the Mini.
Unrelated to the Mac, I'm also looking at getting a large internal hard disk for my SK41G [Silver] Shuttle to host a number of virtual machines, which should now be possible thanks to the 2GB RAM upgrade I put in last week, as well as adding another 1GB to the SN95G5V3 [Black] Shuttle to bring it up to 2GB.
Finally, I'll be looking for a backup mechanism for the Silver Shuttle - maybe another external Firewire drives, but potentially a NAS disk of some sort. All exciting stuff really...
Therefore, the news last week that Apple had upgraded the specification of the Mini was quite exciting, if only to give a bit more confidence that the machine would be around for a while longer.
Following some careful negotiations with T, I will be buying a new Mini when Leopard is released. There is no way that I would buy one now only to have to upgrade the OS in about two months! The addition of the latest iLife 08 will be good, and I'll probably pay out for iWork 08 as well. The latter being very interesting as the new Numbers application provides a new take on the spreadsheet.
There is some uncertainty over whether my Belkin Omnicube will work with it as this has PS/2 ports, and I'm not sure if a USB to PS/2 converter will do the job, so I might need to factor in a KVM upgrade as well.
I'm guessing I'll need to get an external hard disk for the Mini to get the benefit of Time Machine, perhaps something that will match the style of the Mini.
Unrelated to the Mac, I'm also looking at getting a large internal hard disk for my SK41G [Silver] Shuttle to host a number of virtual machines, which should now be possible thanks to the 2GB RAM upgrade I put in last week, as well as adding another 1GB to the SN95G5V3 [Black] Shuttle to bring it up to 2GB.
Finally, I'll be looking for a backup mechanism for the Silver Shuttle - maybe another external Firewire drives, but potentially a NAS disk of some sort. All exciting stuff really...
Tuesday, 7 August 2007
Backing up Vista
Bought some DVD+R discs yesterday and created the HP Recovery Discs for T's new Vista PC. This means the course of system recovery is:
1) Perform factory-setting rebuild using recovery partition
2) If hard drive has died completely, use the Recovery Discs
There is also a bootable Hardware Diagnostics boot disc which could help identify problems. Of course, this won't recover any user data, so for that, Vista has a built-in backup program.
The Backup and Restore Center is different depending on the version of Vista being used. The Home Premium version (T's machine) supports backing up user data files, but not the entire machine. The Business version has the ability to do user data files, but can also backup the entire computer in the form of an image (a la Ghost).
On the Home version, backing up was a breeze. I had attached a 160GB LaCie D2 drive over Firewire (400) which Vista automatically picked up. I pointed the program to the external driveand it prompted me to select a backup schedule. The default was to create a weekly backup every Sunday night which I accepted. The backup kicked off and went and did its stuff and that was that.
Very easy and impressive. Given the amount and type of data people store on a PC these days, the ability to do a backup is essential and for it to be used, the process must be painless. Vista appears to have got this right (of course, the ability to restore is also critical, but that's for another day).
On an unrelated note, my RAM upgrade has arrived for my second Shuttle, so this should see the start of the big virtualisation project soon.
1) Perform factory-setting rebuild using recovery partition
2) If hard drive has died completely, use the Recovery Discs
There is also a bootable Hardware Diagnostics boot disc which could help identify problems. Of course, this won't recover any user data, so for that, Vista has a built-in backup program.
The Backup and Restore Center is different depending on the version of Vista being used. The Home Premium version (T's machine) supports backing up user data files, but not the entire machine. The Business version has the ability to do user data files, but can also backup the entire computer in the form of an image (a la Ghost).
On the Home version, backing up was a breeze. I had attached a 160GB LaCie D2 drive over Firewire (400) which Vista automatically picked up. I pointed the program to the external driveand it prompted me to select a backup schedule. The default was to create a weekly backup every Sunday night which I accepted. The backup kicked off and went and did its stuff and that was that.
Very easy and impressive. Given the amount and type of data people store on a PC these days, the ability to do a backup is essential and for it to be used, the process must be painless. Vista appears to have got this right (of course, the ability to restore is also critical, but that's for another day).
On an unrelated note, my RAM upgrade has arrived for my second Shuttle, so this should see the start of the big virtualisation project soon.
Monday, 6 August 2007
New HP S3150 PC
Yesterday, T and I bought a new PC from our local PC World. A few years ago the thought of doing this would have been unthinkable, but for me personally, the days of building from scratch are over and the extra cost of getting a pre-built machine is negligable.
The model I originally specced went end of life last week, so I had to look for an alternative.
The new machine is an HP S3150 with a Core 2 Duo, 2GB RAM, 250GB hard disk, TV tuner and a 20" HP TFT widescreen monitor. It's not the most powerful gaming rig ever built, but will be perfect for T's needs, and it's got a respectable Windows Experience score.
The S3150 is designed as a media centre PC and is tiny: about 11cm wide and under 30cm high. It fits nicely at the bottom of the PC desk and is quite quiet (but not silent). As a media-oriented PC, it's got a remote control (that I haven't played with yet) and a wireless keyboard and mouse. I've traditionally not been a fan of these, but I've been pleasantly surprised by the ones supplied.
It's running Vista Home Premium and having spent about an hour removing all the pre-installed software that scatters unwanted icons on the desktop (and the obnoxious Norton Internet Security), it's now at the stage where I'm ready to do a proper system backup.
One of the tools provided allows the creation of a hardware diagnostics boot CD which I've done but not tested. It also has the ability to generate some recovery media (to restore the PC back to its factory defaults), and I'm under the impression Vista has it's own tools, but need to investigate these further. I've read that some people have had problems creating recovery media with DVD-RW (or was it DVD+RW) so will need to check this out.
Interestingly, the 250GB is partitioned with a 50GB C: drive, a smaller 8GB(?) recovery partition and the remaining space unpartitioned. This would surely confuse inexperienced, normal users.
I've setup users for me (the default administrator user) and T (a normal user), but I've not installed any anti-virus software yet. I'm intriuged to see what Microsoft OneCare is like and will investigate further.
Also need to work out a regular way of backing up the system for T - probably using an external hard drive, and will need to get a small print server so that we can both share the printer.
All in all, a lot of fun to be had with it.
The model I originally specced went end of life last week, so I had to look for an alternative.
The new machine is an HP S3150 with a Core 2 Duo, 2GB RAM, 250GB hard disk, TV tuner and a 20" HP TFT widescreen monitor. It's not the most powerful gaming rig ever built, but will be perfect for T's needs, and it's got a respectable Windows Experience score.
The S3150 is designed as a media centre PC and is tiny: about 11cm wide and under 30cm high. It fits nicely at the bottom of the PC desk and is quite quiet (but not silent). As a media-oriented PC, it's got a remote control (that I haven't played with yet) and a wireless keyboard and mouse. I've traditionally not been a fan of these, but I've been pleasantly surprised by the ones supplied.
It's running Vista Home Premium and having spent about an hour removing all the pre-installed software that scatters unwanted icons on the desktop (and the obnoxious Norton Internet Security), it's now at the stage where I'm ready to do a proper system backup.
One of the tools provided allows the creation of a hardware diagnostics boot CD which I've done but not tested. It also has the ability to generate some recovery media (to restore the PC back to its factory defaults), and I'm under the impression Vista has it's own tools, but need to investigate these further. I've read that some people have had problems creating recovery media with DVD-RW (or was it DVD+RW) so will need to check this out.
Interestingly, the 250GB is partitioned with a 50GB C: drive, a smaller 8GB(?) recovery partition and the remaining space unpartitioned. This would surely confuse inexperienced, normal users.
I've setup users for me (the default administrator user) and T (a normal user), but I've not installed any anti-virus software yet. I'm intriuged to see what Microsoft OneCare is like and will investigate further.
Also need to work out a regular way of backing up the system for T - probably using an external hard drive, and will need to get a small print server so that we can both share the printer.
All in all, a lot of fun to be had with it.
Friday, 3 August 2007
New Virtual Projects
I've got a RAM upgrade on order for my second Shuttle (currently running Solaris Express). This will give the box 2GB RAM and I'm intending to put a few servers on it using VMWare Server.
Last night I put Fedora 7 on it and was reminded that the onboard graphics chip has a signal problem that results in a slight blur. It also eats into the main RAM which is never ideal.
So I dug out my old Matrox G450 and popped it in the server.
What an amazing contrast! The picture quality of the G450 remains stunningly sharp. Although my main graphics card is fine, the G450 has perhaps a sharper screen quality. I'm not expecting the G450 to have enough power to run XGL for wobbly windows, but that's not the point of the server.
I'm hoping to run a couple of VMs - OpenSolaris or Solaris Express, as well as a hardened Linux install (CentOS 5 or Fedora 7 with SELinux enabled). The ability to host a couple of small Solaris Zones within a VM should be interesting as well and will be useful for revising the Solaris 10 upgrade exam.
Then it will be a case of seeing how much of my own stuff I can access from the cloud. Technologies such as iFolder, FreeNX and some web based IMAP client (Roundcube maybe) will be interesting.
Last night I put Fedora 7 on it and was reminded that the onboard graphics chip has a signal problem that results in a slight blur. It also eats into the main RAM which is never ideal.
So I dug out my old Matrox G450 and popped it in the server.
What an amazing contrast! The picture quality of the G450 remains stunningly sharp. Although my main graphics card is fine, the G450 has perhaps a sharper screen quality. I'm not expecting the G450 to have enough power to run XGL for wobbly windows, but that's not the point of the server.
I'm hoping to run a couple of VMs - OpenSolaris or Solaris Express, as well as a hardened Linux install (CentOS 5 or Fedora 7 with SELinux enabled). The ability to host a couple of small Solaris Zones within a VM should be interesting as well and will be useful for revising the Solaris 10 upgrade exam.
Then it will be a case of seeing how much of my own stuff I can access from the cloud. Technologies such as iFolder, FreeNX and some web based IMAP client (Roundcube maybe) will be interesting.
Thursday, 2 August 2007
Windows Server 2008: Reinventing Unix?
Regular readers (hah!) will be aware that I am very impressed with Windows Vista. One of the reasons for this is that Windows is finally reaching the point where it can be run as a limited user and the system is more tightly enforcing the separation between system and user data (the virtualisation feature in the registry is very impressive). It also forces better practices such as not allowing users to write into the root of the C: drive.
With this in mind, I've been doing some initial reading up on Windows Server 2008, partly because the Microsoft dominance in IT means I'll have to become familiar with it at some point, but also because it looks like it might be a very interesting operating system.
Firstly, the OS can be stripped down to perform a specific role (file server, database server, web server etc) and only the required components are installed. This should result in better security (a smaller surface area to attack) and perhaps better performance (less processes running) which should mean Windows Server 2008 has the potential to be easier to manage.
Secondly, the Powershell is included by default, finally bringing a [best of breed] scripting environment to Windows. Apparently the whole operating system should be mangaged through the command line, which should certainly improve supporting customers over sometimes slow WAN links.
If these two features sound similar to Unix, then Henry Spencer's now classic quote "Those who don't understand UNIX are condemned to reinvent it, poorly." carries some resonance. The fact is that with Windows Server 2008, Microsoft are creating an evolution of their OS that does sound a lot more like Unix than previous versions.
And this can only be a good thing!
With this in mind, I've been doing some initial reading up on Windows Server 2008, partly because the Microsoft dominance in IT means I'll have to become familiar with it at some point, but also because it looks like it might be a very interesting operating system.
Firstly, the OS can be stripped down to perform a specific role (file server, database server, web server etc) and only the required components are installed. This should result in better security (a smaller surface area to attack) and perhaps better performance (less processes running) which should mean Windows Server 2008 has the potential to be easier to manage.
Secondly, the Powershell is included by default, finally bringing a [best of breed] scripting environment to Windows. Apparently the whole operating system should be mangaged through the command line, which should certainly improve supporting customers over sometimes slow WAN links.
If these two features sound similar to Unix, then Henry Spencer's now classic quote "Those who don't understand UNIX are condemned to reinvent it, poorly." carries some resonance. The fact is that with Windows Server 2008, Microsoft are creating an evolution of their OS that does sound a lot more like Unix than previous versions.
And this can only be a good thing!
Subscribe to:
Posts (Atom)