For those who don’t know, there is information about it on wikipedia. Seriously, if you don’t know what raid is, you won’t be interested in this post. Those who do know may be wondering why i went software instead of hardware. Hardware raid is a dedicated hardware device (pci card) that implements raid on board and does all the xor’ing, etc and gives the device to the system as a single drive. Theoretically, this is faster than software and more reliable. And its true…at $1000 a card. Really, anything less and you’re almost certainly doing software raid anyway with the disadvantages of hardware. Namely, if the raid controller dies, and you don’t have the exact same model number and firmware revision, you’re screwed. Turns out that every single card out there stores the raid metadata in a slightly different way and if you can’t find the same card, your data is gone forever. Linux software raid doesn’t have this problem. Just pop in a bootable cd and you have full reliable access to your data. Move your hard drives to a new system/controller? No problem! It all works fine.

The other standard argument for hardware over software is performance. Again its true…at $1000 a card. Its an xor operation. Nothing more, ever. Ok, if you’re main cpu is a pentium II underclocked to 5 mhz then yeah, software might slow things down. At the end of this post i’ll include some info on the overhead i’ve experienced with it.

So, the actual process in linux of creating a raid array. It turns out that, unlike ndiswrapper or sendmail configuration *shudder*, its incredibly easy. Really, just make sure you have your hard drives in, (and partitioned), and then do one simple command like the following and its done.

sudo mdadm --create /dev/md0  --chunk=16 --level=5 --raid-devices=4 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1

Now, before we go any further, lets talk about partitioning. Strictly speaking, its not necessary. Everything will work just as well on a bare drive as on one thats been partitioned. The main reason you’d want to, and the reason i recommend you do, is because not all drives are created equally. When you create a raid, by default, the smallest drive sets the size of all the other drives in the array. So, if you have a 200 gig drive and 2 500’s, the net result will be an array of 3 200’s. The real danger is, drives fail. Often. I can’t even count the number of dead hard drive’s i’ve personally seen working for LCSD2. And if you made an array of 3 500 gig drives and one dies, the replacement has to be at least as big or it won’t work. If the replacement is even 1 byte off, you cannot add it to the array. Now you’ve just wasted money on a drive you can’t use and you run the risk of another drive failure and loss of all your data before you can obtain another (in a raid5). So, i recommend that you partition each drive so that the partition on each is the same size but at least a couple of megs less than the total drive capacity. Sure you lose 20 megs but you gain peace of mind :P.

Ok, back to creating the array. Lets break that command down a little.

mdadm

This is the command. This is the one, (and only one), command that you will be using to manage your array. This is the one that create the array, assemble the array, start the array, stop the array, grow the array, replace/remove drives in the array, and finally, destroy the array (note: be careful with that part).

--create /dev/md0

This command specifies the device that will be your array. This is the one that you will format and mount and add files to. We call it md0 (multi disk number 0). This is mostly out of tradition. For example, i could’ve said:

--create /dev/jessesreallybigfreakingraidarraywithlotsofstoragespace

Yeah, i recommend not doing that and just going with md0, md1, etc.

--chunk=16

Ok, chunk size. Basically, in a raid5, data is striped across all the disks in the array. Chunk size, i’m pretty sure, specifies the size of that stripe before it will be written and needs to be a multiple of 4 (2?). Theoretically (and yeah, empirically), this has an effect on the speed and performance of the array. I would advise against obsessing here and just using 16. I did a lot of research and found only contradictory information. e.g Anything over 128 is wasteful! Or Anything under 128 is a waste! I’ll include links in my references section at the bottom if you really want to obsess. In my testing 16 was basically as good as most anything else. Also remember, you chose raid5 for reliability, not necessarily speed.

--level=5

This is where you specify what kind of raid you want. Me, i wanted raid5. You can specify any raid level you want here. I won’t spend time describing each raid level, for that, wikipedia is your friend. I will mention that chunk size has a different effect on raid0 so i would recommend reading the man pages.

--raid-devices=4 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1

Pretty self explanatory. Here you specify how many total devices are in this array and then list them. Notice how all of them specify a partition?

Now you’re array is created. I can’t speak for the others but, if you chose raid5, the array is created with one drive “missing” and it then “rebuilds” itself. It does this for performance reasons (and yes, it really does). The array will be a little slower till its done rebuilding but its fine. Go ahead an use it all you want (or be paranoid and wait, whatever you have time for). It took 3-4 hours for it to completely rebuild for my array. Next step, formatting :).

mkfs.ext3 /dev/md0

There, done. Its formatted. Next.
Ok, fine. There are a few little things you can do here to make things a little more efficient. Also, you don’t have to use ext3 as the filesystem if you don’t want to. With ext3, you can specify things like stride length and width which will try to optimize things to match the block size, etc. to try and make it so that fewer read/write operations are needed. As with block size, i didn’t bother. In my (limited) testing, i found it to not make enough of a difference to warrant my doing anything. If you want to try, i have links below.

Ok, now make a mount point somewhere, for example:

mkdir /mnt/raid

And add the device to your fstab so that its mounted automatically at boot.

/dev/md0    /mnt/raid     ext3    defaults,errors=remount-ro,noatime,noexec,nodev    0    2

One thing to note here. In ubuntu, the mere installation of mdadm causes a new init ramdisk to be created, etc. and basically, it automatically scans for arrays on boot. If you’re distro doesn’t do that, you need to figure out to make or add the following to your startup scripts.

mdadm --assemble --scan

Also, you’ll probably have problems auto-mounting because your array may not be available when the fstab is processed. Let me know if you have problems and i’ll try to help.

• Tips

The whole point of raid5 is redundancy. The idea being that the loss of a single drive means nothing because you can still get to your data and you can replace the drive. However, if you are one drive short on boot, the array won’t be automatically started (at least on ubuntu, i’m not sure this is standard behavior). To force an array to start sans a drive, do the following

sudo mdadm -R /dev/md0

Now you can backup data (highly recommended if possible), shutdown, pop the bad drive out, put the shiny new drive in, and add it to the array.

sudo mdadm --add /dev/md0 /dev/sde1

Reconstruction begins automatically. You can follow the progress by check /proc/mdstat periodically.

cat /proc/mdstat

This will take a long time, especially if you just let it go. By default, it limits the speed of the reconstruction to leave room for normal operation. If you have a mostly idle array or are just impatient, you can increase the speed limit. The speed limit is stored in “proc” virtual filesystem.

cat /proc/sys/dev/raid/speed_limit_min

By default, the speed limit is 1000 kb/s. Thats very slow, not even a megabyte a second. On a large array, it could take days at that speed. To increase the speed, just overwrite the file (as root, sudo won’t work here).

sudo -s
echo 15000 > /proc/sys/dev/raid/speed_limit_min

This increases the minimum speed limit to around 15 mb/s. Much better. Keep in mind that just because you make it that high, it might not reach the speed. It is reading and writing constantly at the same time to rebuild that drive and can only go as fast as that drive (and bus) can handle. That said, this is safe. There should be no danger in increasing the speed and a simple reboot will return it defaults. Also, it should be mentioned that a clean reboot (not a power failure probably) is fine and reconstruction will resume when it comes back up.

Now its been a while and we’ve ran out of space on our array. We don’t want to start over with bigger drives but we have space for one more. All we have to do is tell the array it has another drive and should “grow” to encompass it. No problem.

sudo mdadm --add /dev/md0 /dev/sdf1
sudo mdadm --grow /dev/md0 --raid-devices=5

This will take a while and you may want to use the speed limit trick above to speed things up. Now the array is bigger but theres a problem, you still don’t see the extra space. The reason is the filesystem is still the size of the old array. You need to resize the filesystem to accommodate the extra space. This is also easy.

sudo e2fsck -f /dev/md0
sudo resize2fs -p /dev/md0

Resize2fs complains if you don’t do a file system check first so thats the reason for that. This will only take a minute or two and you will have the full space available to you. Technically its possible to do this online (while the filesystem is mounted) but i recommend unmounting it first. This command also only works on ext3 filesystems so you’ll need to find the appropriate command for your file system of choice.

If you ever want to check the status of your array, you can just look at the mdstat file.

cat /proc/mdstat

This will show you the arrays it knows about and their status. For more information you can use mdadm

mdadm -D /dev/md0

You can also use mdadm to monitor the array.

mdadm --monitor

This command starts mdadm and forks off a daemon that keeps an eye on the arrays. If an array changes its state say, when a drive dies, mdadm will send off an email about it. The configuration file is in /etc/mdadm/mdadm.conf. Ubuntu by default has this running when its installed and will send an email to root when something happens.

I’ve had a really good experience with my array. I’ve tested it thoroughly and fully trust it. Its rock solid. I’d like to know about your experiences, leave a message or a link in the comments. Also if you need any help, i’d be happy to try.

• References

to be added.

In Mac OS X there are at least two methods for resetting a password. And it does require resetting. This isn’t windows, you can’t just boot off a cd and automatically crack the password. Apple decided to used modern encryption techniques 10 years ago that were probably 10 years old then to protect your passwords. Microsoft still hasn’t.

Method number 1: Follow the instructions here to boot off a disc and reset your password the apple way.

Method number 2: Follow my instructions below and reset it the “cool” way.

Before i begin, there’s something i should talk about. In the traditional *nix world (unix, linux, solaris…nix), the idea of storing passwords in a flat file on the filesystem is the norm. Usually, in linux anyway, user info is in /etc/passwd (user id, shell, name, etc.) and the password data is encrypted in /etc/shadow. You will notice in OS X that the file /etc/fstab exists. So that file is where all user account data is stored, right? WRONG! This file exists but it is only consulted/used in any way in single user mode (i’ll get to that later). The thing to take away here is that in OS X, account data is stored in a database. Prior to 10.5 (maybe 10.4 too??) this database was apple’s proprietary netinfo database. In 10.5, netinfo was done away with completely and replaced something else ( i think its openldap). Either way, theres a little more to it than editing a text file.

I mentioned single user mode before. In the *nix world (again. unix, linux, etc.), there is some concept for allowing access to the system without loading the entire system up. You know, in case of emergencies or for maintenance reasons. Microsoft actually does this also with the recovery console…kinda. The important thing is, (unless you’ve locked it down) it gives you access to the system without authenticating. Whats more is that it gives you “root” access.

Ok, lets get to it. To enter single user mode, reboot the machine and, at the chime, hold down the “apple” and the “s” keys. Keep holding them down until you are presented with a text only prompt. We will be modifying files on the filesystem so we need to remount the filesystem as read/write. Type the following:

mount -uw /

Next step, we need to load the daemon responsible for providing access to the account database. This is a little different between 10.4 and 10.5. In 10.4, despite my sincerest efforts, i was unable to find a reliable way of starting only the process needed. I’ve only had trouble on clients 10.4.9 and below and i’m pretty sure its an apple bug. It just doesn’t work right (hangs at blue aqua screen). If you’re using 10.4.9 or below, good luck or check the end of this document for an alternative.

10.4:

sh /etc/rc

In 10.5, apple will actually give you the command to use if you try to use dscl (directory services command line utility).

10.5:

launchtl load /System/Library/LaunchDaemons/com.apple.DirectoryServicesLocal.plist

Now the daemon is running and you have access to the database. What we’re going to do here is enable the root account. Remember how i said that you currently have root access? Well, you can’t type passwd, give it a password and be done. The reason is like i said, right now /etc/passwd is being consulted. At best the command would fail. At worst, nothing would happen. You need to change the directory database root user’s password. I’m pretty sure this is the same in 10.4 and 10.5. We’ll be doing this interactive mode:

dscl .

This opens the command line utility and gives you “shell” access to the database. Next, we navigate to the right user like so…

cd /Users

Pretty easy so far. Now we’ll give the root account a password thus enabling it on boot. You can also substitute “root” here for any other account you see after typing ls to reset that password.

/Users > passwd root
New Password:
/Users >

Just enter your password of choice after “New Password” and your done. Type exit and then reboot to reboot into normal mode

exit
reboot

When the computer comes up, either click on other and type root for user and the password you gave or just type it in the name a password fields and voila! Superuser access. Now be careful. “With Great Power Comes Great Responsibility”. Seriously, the superuser isn’t inhibited by any permissions or many of the security features of the system. You can cause real damage and not even be warned about it with him. That said, you can also use it to ignore permissions and retrieve files or whatever you need to do.

Method 2.b. Make your own account.

Maybe you need into the computer but don’t want to bother any account data already there. The following commands, in non-interactive mode, will create a user called Administrator with administrative privileges.

dscl . -create /Users/"admin"
dscl . -create /Users/"admin" UserShell /bin/bash
dscl . -create /Users/"admin" RealName "Administrator"
dscl . -create /Users/"admin" UniqueID "id"
dscl . -create /Users/"admin" PrimaryGroupID "20"
dscl . -create /Users/"admin" NFSHomeDirectory /Users/admin
dscl . -passwd /Users/"admin" "password"
dscl . -append /Groups/admin GroupMembership admin

One thing about the above. The UniqueID. This has to be unique. One thing you can do to determine a usable uniqueid is to run this command:

echo $[$(dscl . -list /Users uid | awk '{print $2}' | sort -n | tail -n1)+1]

Of course, you’re probably safe using a high number like 550 provided you don’t have 50 or more local users on your machine. You can change the commands above to create a user with a different name, etc. but be careful. Theres a little more to it than that. Here are apples instructions on creating users this way but they leave out creating a group for the user and just using staff instead. Thats probably fine but i’ve had trouble not creating a group before. I may do a writeup on that later.

Now, the security conscious people out there are probably wondering, “isn’t this a huge security hole? I mean anyone can gain root whenever they want”. First, no they can’t. This might sound obvious but to do any of this, they have to be sitting in front of the machine. For years, security experts have said physical security is the first step to a secure system. Seriously, any machine can be compromised if you have physical access to it. FWIW, there are any number of tools available online to reset windows passwords. This same basic technique can also be applied to any linux system that hasn’t been really locked down but again, with physical access, you can reset it. That said, you can’t always have physical security. In a school for example, kids may be in a position to reboot a computer and do this without a teacher really noticing. If you want to, there are a few things you can do to mitigate the problem.

1. Set a firmware password. Here are apple’s instruction for doing so. The upside is it prevents both single user mode and booting from a disc without the password. The downside however is once again, physical security. If you have physical access to the machine you can reset the firmware password by removing some memory, turning it on and off, and putting the memory back in. This is, however, much less easy to do in a school lab environment.
2. Turn on filevault. Filevault will encrypt your home folder and all files in it using aes encryption. Filevault requires your account password to decrypt it or alternatively, the master password. The master password cannot be reset because only the one that encrypted it can decrypt it. This secures your data but a new account could still be made allowing someone to use the machine.
3. Disable single user mode. I hesitate to mention this because people might think its a legitimate thing to do. Don’t. Unless you’re sure you know what you’re doing, don’t. I’m not going to post instructions or links here but a quick google search should give you what you want.

Alternative method.

There is another thing you can do. If you don’t care about the current user account or its corrupt or something, you can reset it completely. Understand that all your login credentials will be wiped out and you will have to create a new one. First, enter single user mode and mount the filesystem read/write. Then do this:

10.4

rm -Rf /var/db/netinfo

10.5

rm -Rf /var/db/dslocal
rm /var/db/.AppleSetupDone

This removes the database files and forces Mac OS X to go through the setup assistant again allowing you to create an administrative account as though it were a brand new machine

Well, i hope this article has been informative. Please leave a comment if you like or have any suggestions/corrections to make.

*Update. Turns out i was a little off on (at least)one thing. In Mac OS X 10.5, netinfo has been completely dropped in favor of xml based files /var/db/dslocal. It looks like the directory services daemon starts at runtime and reads these in allowing for database style queries to it.

*Update 2. I really need to learn to proofread. Corrected many spelling errors and added commands where i had previously left placeholders. Enjoy.

*Update 3. Mac (seriously “mac”? thats the best you could do?) has pointed out that i’ve made a mistake. In 10.5 there is an additional step to take after deleting the directory database. I’ve added it above.

Now the reason i wanted drop down menu’s in the first place is for semantic organization. Hover over about me and you get contact info, etc., stuff that belongs under about. It also allows me to have more static content while still using this theme. I rather like this theme/don’t feel like looking for another and would prefer to keep it. Now i can.

Sometime in the future i will be adding things like contact info and actual content. If you have any suggestions/hate, please use the contact page to let me know.

Now, first things first; you need to run your own dns server. Secondly, the target must be using your dns server for lookups.

The first thing we need is the ip address to kittenwar.com. Thats easy. Its 64.111.96.38. I shouldn’t have even mentioned this part.

Secondly, we need to construct a zone declaration that will claim to be the master for anything. I used the following:

zone "." {
	type master;
	file "kittenwar/all.hosts";
	};

It worked great. See, in dns, all domains end with an usually implied “.”. (Its the one between the quotes) This dot represents the root nameservers and how they’re the start of the domain name system. I claim here to be the master authoritative namerserver for any domain ending in “.”. That is to say, all domains.

Next, is the actual redirection…trickery…dns spoofing…whatever. We now need to match all domains to have an A record of 64.111.96.38. Luckily, this is easy. I used the following:

$TTL	86400
@			IN	SOA	localhost. root.localhost. (
					      2		; Serial
					 604800		; Refresh
					  86400		; Retry
					2419200		; Expire
					  86400 )	; Negative Cache TTL
;
@			IN	NS	localhost.
@			IN	A	64.111.96.38
*			IN	A	64.111.96.38

Which also worked great. Most of that isn’t strictly necessary for a zone like this but, i already had that one mostly made. As you can see, we define “.” to have an A record for kittenwar as well as anything else “*”. Well thats great, your done, your target is now going to kittenwar.com and there’s nothing he can do, right? WRONG!! You still need to select that victim to be the one that feels your wrath.

This next step involves views. If your not using views, well, you should. The tricky thing about them is every zone *must* be in a view if even one is. So, remember that. Bind cries bloody tears of pain everytime you don’t. Sorry just wanted to make that point as graphic as possible. You can add them manually if you want. The way i did it, was with acl’s. I used the following:

acl kittened { 192.168.0.102; 192.168.0.110; };

This matches both my brother (192.168.0.102) and myself (192.168.0.110). I added myself so i could test that its working. I recommend doing this temporarily. Now, add the acl to any views that the hosts may already be matched by.

view "internal" {
    match-clients { !kittened; localnets; };
    recursion yes;
    include "/etc/bind/internal/internal.conf";
};

There i’ve removed the acl “kittened” and all its hosts from normal service because i’m going to add them to a completely separate view. Now, lets create that view. The syntax is fairly simple, really just copy and paste from above where necessary.

view "kittenwar" {
    match-clients { kittened; };
    recursion no;
    include "/etc/bind/kittenwar/kittenwar.conf";
};

And thats it. I turned off recursion for this view because its already authoritative for everything. You don’t need to. Give bind a good restart (it tends to like restarts better for changes like this) and try it out. Now any domain/website that the target attempts to visit will simply bring up the page kittenwar.com.

*Note. Operatings systems and browsers both tend to cache dns lookups. It may take some time for these to expire and the effects to be seen. For best results, have ssh access to the machine and clear the cache yourself/reboot the machine. Also, you can do this for any site for which visiting the ip address takes you to the site. Thats not always true, especially with shared hosting. If you have any questions at all, leave a comment.

*Note 2. When the target goes insane and begins chasing you around the house with a bloody chainsaw (why do the chainsaws always already have blood on them?). And it will happen. Its very easy to reverse this. Just remove the ip address from the acl declaration and restart bind. Then restart the machine/clear caches. Intense “The Shining” style killing urges should subside after a while. If it doesn’t, use your telepathic powers to summon the black guy to his death. Somehow thats relevant and everything ends with you living despite being a retarded 9 year old vs a madman with an axe.

I like managing a webserver. I like manually managing bind and apache from the command line. I like the sense of control i feel when i am able to make any changes i want any time i want. What i don’t like though, is email. Email is evil. Go ahead, take a look at the configuration files for sendmail sometime. Try postfix, not much better. I don’t want to do email but i want to get email. So what did i do? I went with google apps for your domain.

Google apps for your domain is a godsend. I make a few small changed to my dns, which i can easily do; specifying google as my mail exchanger (mx) and i’m done. Below i’ve included the records required for email.

@           IN    MX 1     aspmx.l.google.com.
@           IN    MX 5     alt1.aspmx.l.google.com.
@           IN    MX 5     alt2.aspmx.l.google.com.
@           IN    MX 10    ASPMX2.GOOGLEMAIL.COM.
@           IN    MX 10    ASPMX3.GOOGLEMAIL.COM.
@           IN    MX 10    ASPMX4.GOOGLEMAIL.COM.
@           IN    MX 10    ASPMX5.GOOGLEMAIL.COM.

Google handles all my email for me. I can use their nice, friendly-ish web interface to add up to 100 users for email. I can add mailing lists so that an email sent to, for example, lists@jessecole.org would go to whoever i added to that list. I can tie in my other domains with it so that jesse@jessecole.info works just as well as jesse@jessecole.org and it gets delivered to the same mailbox. On top of all of this, i get google’s fantastic! email filtering. I never get a single spam message through and very rarely do i see a false positive. Google even provides pop, smtp, and imap access to my email.

Another feature that isn’t talked about so much is google talk. Thats right, every user i make can log in to google talk with user@domain.com. The can also use any jabber client to log in. Also, i recently implemented XMPP server dialback as per section 14.4 of RFC 3920 allowing virtually all jabber clients to talk to anyone logged in. Ok, fine. All that is accurate but i just followed google’s instructions here. Incidently, its perfectly legal in dns-ese to paste the following into your zone file or include it globally.

_xmpp-server._tcp   IN SRV  5 0 5269 xmpp-server.l.google.com.
_xmpp-server._tcp   IN SRV 20 0 5269 xmpp-server1.l.google.com.
_xmpp-server._tcp   IN SRV 20 0 5269 xmpp-server2.l.google.com.
_xmpp-server._tcp   IN SRV 20 0 5269 xmpp-server3.l.google.com.
_xmpp-server._tcp   IN SRV 20 0 5269 xmpp-server4.l.google.com.

_jabber._tcp        IN SRV  5 0 5269 xmpp-server.l.google.com.
_jabber._tcp        IN SRV 20 0 5269 xmpp-server1.l.google.com.
_jabber._tcp        IN SRV 20 0 5269 xmpp-server2.l.google.com.
_jabber._tcp        IN SRV 20 0 5269 xmpp-server3.l.google.com.
_jabber._tcp        IN SRV 20 0 5269 xmpp-server4.l.google.com.

Fully qualified domain names are for quitters.

With google apps for your domain you get all this and all of the google apps. Thats right. Your <employee>@jessecole.org account also gets its own google docs, google calendar, google everything!. Its no wonder some businesses are using for their site. The best part about all of this is that its free! You can pay extra and get more users, features, support, etc. Not necessary. This is a great thing for little websites like me.

I hope you enjoyed my little post (post? this is an article!) on google apps. Please, leave me a comment if you liked it/disagree/hate me/whatever.

*Update* I am officially declaring the whitespace stripping in pre text a wordpress bug. I think i’ll actually file one later today. It won’t do it if you step just right through the posting process though so it should look good now.

Quarterly report 2008-01

http://www.jessecole.org
Total uptime:99.13% Downtime:6 hour(s) 13 min(s)
Quarterly uptime:99.13% Downtime:6 hour(s) 13 min(s)
Month 2008-03 Uptime:99.13% Downtime:6 hour(s) 13 min(s)

*Update*So, wordpress helpfully stripped the white space. Yay!

I hope you enjoy the new changes and happy blogging :).

public class CountPrime
{
public static void main(String[] args)
{
for(long i=1;i<30000000;i=i+2)
{
for(long j=i-1;j>1;j--)
{
if(i%j==0) break;
if(j==2) System.out.println(i);
}
}	}
}

This bring me to a little mini-rant. Notice how i’m using longs in the code above? Why not just do something smart and use unsigned integers giving you a nice 2^32 long space to work with? Well here’s the answer. JAVA DOESN’T FREAKING SUPPORT UNSIGNED ANYTHING!!!! I took 2 classes on java and never heard a word about this. I’ve defended java from idiots that for some reason like c or c++ without knowing this. How do you freaking design a programming language without support for something so unbelievable basic as unsigned integers or short or chars or bytes or longs or anything like that?

Ok, that was a little more than a mini-rant but i feel better now. Anywho, here are the list of prime numbers. The file is tar’ed and gzip’ed so if you’re using the only os in the world that doesn’t support it built-in (windows), its time to upgrade to something ‘nix based.