For all who don't know what mdadm is, mdadm is a software RAID tool for Linux. Now we've got that out of the way...
As any other Über-geek I have a Linux server running some software and storing all of my precious data and of course my software repositories. As I have had the pleasure of data loss a couple years back I decided to use RAID 5 for my server. Of course I'm a student (at the time writing), and I don't have a lot of money because I'm not a greedy bastard. :-) I of course went for a software RAID using mdadm as my weapon of choice.
My server is an old desktop computer (I won't name the brand, but can ensure you it is notorious for.. well.. (since there is no other way to put this) sucking). After infecting the PC with penguins I found the server pretty damn good. Lucky me :-D
The *cough* server *cough* has a AMD Athlon 64 X2 3800+, 2GHz isn't too bad for downloading crap, serving Redmine and some software repositories like SVN and Mercurial. (No GIT, I should be ashamed.) It came with IIRC 1GB of RAM which I naturally upgraded to 2GB as I had 1GB laying around. After stripping some useless hardware like a graphics card I popped in a cheap-ass soft-RAID card and 4x500GB, you can guess the rest.
After some time I wanted to upgrade the thing as I had another system that I didn't use. The upgrade would consist of dual core to quad core, 2GB to 5GB (2x2GB 2x512MB) and 4x500GB to 4x1TB.
The memory upgrade is successful and the CPU upgrade didn't work as the motherboard didn't support the quad core.
The hard drives would be tricky as it would consist of yanking out one drive at a time and replacing them with a bigger drive. As mdadm f*ckt me royally in the past I decided to backup my RAID first. I had an external drive of 1TB laying around and used rsync to create a full backup (yeah I have a sh*tload of hardware I don't actually use but comes in handy sometimes).
Today I spent a couple hours of upgrading and soon it was time for the big drive swap. Mdadm reported earlier today that one of the drives failed, so that one was first to be replaced. Using hdparm to check the serial of the drive I recognized the drive and replaced it with a 1TB disk. After running "mdadm /dev/md0 --add /dev/new-drive" it started recovery. Of course this is where mdadm screwed me over again and showed me something like "[U_U_]" or "[_UU_]". (U means that a drive is Up and _ means a drive is down, or something.)
As I have a ramdisk (initramfs, initrd, etc.) with mdadm build in and busybox as a rescue shell I tried to force mdadm to assemble the array. Assembly didn't work so I tried re-creating them (keeping the data) by issuing "mdadm --create /dev/md0 --raid-devices=4 --level=5 --assume-clean /dev/sd[abc] missing" this worked.. until I invited /dev/sdd to the party (what a party pooper).
I am now creating a RAID 5 consisting of 4x1TB using mdadm (I guess I never learn). I guess I'll be restoring backups somewhere tomorrow.
Lesson learned: never trust mdadm! (or make backups before screwing around with mdadm)
maandag 19 november 2012
dinsdag 9 oktober 2012
Starting development UML tool
Today I decided to start development on an idea I had a couple months earlier. I'll start with a quick intro.
The idea is to develop the ultimate UML tool. A free to use (at least for non-profit and education) UML tool capable of creating a wide range of tightly entwined diagrams. Of course I did some research about tools that are already available. A commonly used tool is Visual Paradigm. Visual paradigm supports a wide range of diagrams and is quite easy to use. However it is a tool that quickly loses great functionality when using the free version. I'm also (ironically as a Java developer) not too fond of Java based desktop applications. I find such applications to feel 'heavy'.
There are more UML applications available (Umbrella, BoUML to name a few), but they support only a couple of diagrams and are not (in my opinion) ready for professional use.
The application I'm developing is currently named OUDS (Open UML Documentation standard, see my earlier post about it), but it needs another name for obvious reasons.
I chose Python as my weapon of choice to develop the application. Python has high performance, is cross platform and seems simple enough. For the GUI I picked wxPython. wxPython seems to provide a native look and feel, so my users will feel home right away. :-)
As each software project starts with requirements gathering I thought I'd start off with creating a solution for this. Obviously the requirements need to be stored in some way. Luckily the guys at omg.org had a (sort of) solution available. Requirements Interchange Format (ReqIF) provides a standardized way to store requirements, just what I needed. I also found a page on the Eclipse site stating plans to use this format for UML tools.
As I am quite new to Python this should be a nice experiment. :-)
The idea is to develop the ultimate UML tool. A free to use (at least for non-profit and education) UML tool capable of creating a wide range of tightly entwined diagrams. Of course I did some research about tools that are already available. A commonly used tool is Visual Paradigm. Visual paradigm supports a wide range of diagrams and is quite easy to use. However it is a tool that quickly loses great functionality when using the free version. I'm also (ironically as a Java developer) not too fond of Java based desktop applications. I find such applications to feel 'heavy'.
There are more UML applications available (Umbrella, BoUML to name a few), but they support only a couple of diagrams and are not (in my opinion) ready for professional use.
The application I'm developing is currently named OUDS (Open UML Documentation standard, see my earlier post about it), but it needs another name for obvious reasons.
I chose Python as my weapon of choice to develop the application. Python has high performance, is cross platform and seems simple enough. For the GUI I picked wxPython. wxPython seems to provide a native look and feel, so my users will feel home right away. :-)
As each software project starts with requirements gathering I thought I'd start off with creating a solution for this. Obviously the requirements need to be stored in some way. Luckily the guys at omg.org had a (sort of) solution available. Requirements Interchange Format (ReqIF) provides a standardized way to store requirements, just what I needed. I also found a page on the Eclipse site stating plans to use this format for UML tools.
As I am quite new to Python this should be a nice experiment. :-)
vrijdag 30 maart 2012
Javadoc confusion
As a programmer you often need to look into the documentation. This is of course necessary to know what the code does and how to properly use supplied code. Expecting proper documentation is one of the 'high' standards I assume is provided in code. After all communication is the most important aspect of collaboration.
Javadoc is the standard when writing Java code. A proper Javadoc should look like this:
To my amazement I found this in the Javadoc supplied by Oracle:
As a bonus I'm going to leave this here, just something I found when reviewing code as an assignment for school. I just didn't know whether to laugh or to cry.
Javadoc is the standard when writing Java code. A proper Javadoc should look like this:
/**Everything should be documented so one doesn't have to look at the source code to know what the code does.
* Generate an informational page of this class in the supplied format.
* Valid formats are "html" and "plain".
* @param format The format to create the page in.
* @throws FormatUnknownException If the format supplied by "format" is unknown.
*/
public Page createInfoPage(String format)
{
...
}
To my amazement I found this in the Javadoc supplied by Oracle:
Really? This is not useful at all. How am I supposed to know how to prevent this exception or how do I present this to my users? "Something went wrong?"
- Throws:
NamingException- if a naming exception is encountered
As a bonus I'm going to leave this here, just something I found when reviewing code as an assignment for school. I just didn't know whether to laugh or to cry.
/**
* Returns a short description of the servlet.
*/
public String getServletInfo() {
return "Short description";
}
maandag 6 februari 2012
Hosting multiple Trac environments
Trac is in my opinion a great tool to manage projects. Although most Trac installations are aimed at software development projects this isn't necessary. Trac can be a great tool to manage non-software-development projects too.
I use Trac for almost any project and so I have have a server running with several instances of Trac. One instance per project. The different Trac installs I manage are accessed through a subdomain, trac.example.org for example. Each Trac instance is located in a virtual directory, e.g. trac.example.org/trac1, trac.example.org/trac2, etc.
For some projects I use subversion. I configured apache to serve subversion through DAV. The subversion repositories are accessible using svn.example.org as subdomain and a virtual directory per repository, similar to Trac.
Now as I am the only user of these projects I used a messy configuration that led to some sort of single sign on. Now I've reached the point that more people are going to use Trac and SVN, so the single sign on is no longer desirable (except for me, the administrator). So I came up with a solution to let apache determine access at directory level and let Trac manage its own permissions.
In my apache's "vhost.d" directory I've put a file named trac.conf. This configuration file looks like this:
In the "vhost.d" directory I have a directory named "trac" containing config files that create a trac instance.
example:
The Trac hosting is done with this. The SVN hosting is done by a module. The module is located in apache's "modules.d" directory and it's named "47_mod_dav_svn.conf".
Mine looks like this:
Now I can define access for SVN per repository using a ACL file. Example:
The file defines 3 groups:
prjA only has read/write access to the project-A repository.
prjB only has read/write access to the project-B repository.
note: You can do more things like add a group to a group.
To revoke all permissions in a directory simply specify the user or group followed by a =.
For example if I would like to revoke all permissions for "foo" I'd write a line "foo = " under the directory I want.
The file for the Trac environments is likewise. The difference is that in stead of "[project-A:/]" I would have to specify "[/project-A]" to configure the directory.
Only read access is required as the rest of the permissions depend on Trac's configuration.
The only caveats are that if you use a global config/policy file for all Trac environments and if a user has access to multiple Trac environments, the user has the same permissions in all environments. Luckily this doesn't create a problem for me.
For some projects I use subversion. I configured apache to serve subversion through DAV. The subversion repositories are accessible using svn.example.org as subdomain and a virtual directory per repository, similar to Trac.
Now as I am the only user of these projects I used a messy configuration that led to some sort of single sign on. Now I've reached the point that more people are going to use Trac and SVN, so the single sign on is no longer desirable (except for me, the administrator). So I came up with a solution to let apache determine access at directory level and let Trac manage its own permissions.
In my apache's "vhost.d" directory I've put a file named trac.conf. This configuration file looks like this:
<IfDefine TRAC> #only run this file if TRAC is defined
<IfModule !dav_svn_module> #load dav_svn_module if it isn't loaded already (required by authz_svn_module)
LoadModule dav_svn_module modules/mod_dav_svn.so
</IfModule>
<IfModule !authz_svn_module> #load authz_svn_module if it isn't loaded
LoadModule authz_svn_module modules/mod_authz_svn.so
</IfModule>
Listen 443 #listen on port 443 (SSL)
NameVirtualHost *:443
<VirtualHost *:443>
ServerName trac.example.org #hostname, just an example
Include /etc/apache2/vhosts.d/ssl.include #include some ssl stuff
Include /etc/apache2/vhosts.d/trac/*.conf #include trac projects
Alias /[^/]+/chrome/common /var/www/trac/common #map common trac libs
<Location />
SVNPath /var/www/trac #directory containing Trac http dirs
AuthzSVNAccessFile /var/www/trac/access #policy file
Require valid-user
AuthType Digest
AuthName "/" #must match the realm in the digest file
AuthUserFile /var/www/trac/trac.htpasswd #the digest file
Order deny,allow
SSLRequireSSL #because I want things to be safe :-)
</Location>
</VirtualHost>
</IfDefine>
In the "vhost.d" directory I have a directory named "trac" containing config files that create a trac instance.
example:
<IfDefine TRAC>This file only maps the virtual directory to the Trac wsgi script, generated by trac-admin.
WSGIScriptAlias /project1 /var/www/trac/project1/cgi-bin/trac.wsgi
<Directory /var/www/trac/project1/cgi-bin>
WSGIApplicationGroup %{GLOBAL}
Order deny,allow
Allow from all
</Directory>
</IfDefine>
The Trac hosting is done with this. The SVN hosting is done by a module. The module is located in apache's "modules.d" directory and it's named "47_mod_dav_svn.conf".
Mine looks like this:
<IfDefine SVN> #only host when SVN is defined
<IfModule !dav_svn_module> #load dav_svn_module
LoadModule dav_svn_module modules/mod_dav_svn.so
</IfModule>
<IfDefine SVN_AUTHZ>
<IfModule !authz_svn_module> #load authz module if desired
LoadModule authz_svn_module modules/mod_authz_svn.so
</IfModule>
</IfDefine>
<VirtualHost *:443> #again only listening on SSL
ServerName svn.example.org
Include /etc/apache2/vhosts.d/ssl.include #include common ssl stuff
<Location />
DAV svn
SVNParentPath /var/svn #directory containing the repositories
AuthType Digest
AuthName "/" #matching the digest realm
AuthUserFile /var/www/trac/trac.htpasswd #same digest file as the one used for Trac
AuthzSVNAccessFile /var/svn/access #access file for all SVN repositories
Order deny,allow
Require valid-user
SSLRequireSSL #require SSL :-)
</Location>
</VirtualHost>
</IfDefine>
Now I can define access for SVN per repository using a ACL file. Example:
[groups]
admin = admin, siebz0r
prjA = foo
prjB = bar
[/]
@admin = rw
[project-A:/]
@prjA = rw
[project-B:/]
@prjB = rw
The file defines 3 groups:
- admin
- prjA
- prjB
prjA only has read/write access to the project-A repository.
prjB only has read/write access to the project-B repository.
note: You can do more things like add a group to a group.
To revoke all permissions in a directory simply specify the user or group followed by a =.
For example if I would like to revoke all permissions for "foo" I'd write a line "foo = " under the directory I want.
The file for the Trac environments is likewise. The difference is that in stead of "[project-A:/]" I would have to specify "[/project-A]" to configure the directory.
Only read access is required as the rest of the permissions depend on Trac's configuration.
The only caveats are that if you use a global config/policy file for all Trac environments and if a user has access to multiple Trac environments, the user has the same permissions in all environments. Luckily this doesn't create a problem for me.
woensdag 1 februari 2012
UML as text?
Since I'm working a lot with UML these days I found the need for an open source UML tool that works on Linux and could do more than just a class diagram or a sequence diagram.
Some tools I tried were StarUML, Visual Paradigm (which I use for my education), Umbrello and Dia. All these tools have their good and bad points.
As a die hard Linux user I always try to avoid Windows based applications for obvious reasons, so my experience with StarUML wasn't really good. I believe that if you're a Windows user It's a great tool, but as I said I was looking to do more then a hand full of diagrams.
Umbrello is a neat looking tool, QT based (which I like) and is cross platform. Too bad Umbrello doesn't support all the diagrams I want it to. Umbrello also seemed to crash quite often and it sometimes didn't behave to my liking. (resizing classes in class diagrams made text in the class jump out of the class, etc).
Dia is a well known diagram tool which I use quite often to draw a quick diagram. Unfortunately Dia is a very simple tool and it isn't flexible enough (or it is too flexible). Diagrams need a lot of work to keep them clean. Linking to other diagrams isn't supported and as I hate using a mouse, Dia isn't my tool of choice. I might as well use Microsoft Visio. :-P
Visual Paradigm suited my needs quite fine. It's a great tool if you want to document a lot of diagrams and it also has some neat features like reverse engineering. Linking to other diagrams within Visual Paradigm is allowed and this is super handy. Visual Paradigm however is based on Java (which is a fine language to code in, but doesn't run smooth on my Linux laptop). The user interface is one big mess and the automatic update feature is super annoying, as is the message box in the bottom of the screen. (message box takes literally half the screen on a default install) A lot of features and options I never touched or couldn't find because the UI isn't quite self explaining. The most annoying thing about Visual Paradigm is that it requires an Internet connection to check the license key, each time you start the damn thing. As I am on the road a lot this forms a problem quite often.
Visual Paradigm also comes in two flavors, a paid (Professional, Enterprise, etc) version and a free (community) version. The school I go to provides me with a professional edition so I can create almost all diagrams. However the community edition only allows to create a couple of these (class diagram, sequence diagram, etc.).
Today I wanted to create a diagram and so I fired up Visual Paradigm, it was then I discovered the license key was expired. This left me with yUML.
yUML is an on-line tool that allows you to create a couple of diagrams from text! It then generates an image for you. This is when I came up with an idea that could possibly be quite helpful. A textual representation of UML that can be used as a standard. I'm calling this project "OUDS", Open UML Definition Standard.
OUDS is going to be a way to define any diagram as text. This way a set of libraries can be made in multiple language that can work with an open standard. On top of those libraries a developer can create a GUI tool or anything that would be handy. As a developer I find it my task to take a look at this and if it's possible, create a set of libraries so developers can import, export, parse and do other kinds of neat things.
Some tools I tried were StarUML, Visual Paradigm (which I use for my education), Umbrello and Dia. All these tools have their good and bad points.
Visual Paradigm also comes in two flavors, a paid (Professional, Enterprise, etc) version and a free (community) version. The school I go to provides me with a professional edition so I can create almost all diagrams. However the community edition only allows to create a couple of these (class diagram, sequence diagram, etc.).
Today I wanted to create a diagram and so I fired up Visual Paradigm, it was then I discovered the license key was expired. This left me with yUML.
OUDS is going to be a way to define any diagram as text. This way a set of libraries can be made in multiple language that can work with an open standard. On top of those libraries a developer can create a GUI tool or anything that would be handy. As a developer I find it my task to take a look at this and if it's possible, create a set of libraries so developers can import, export, parse and do other kinds of neat things.
dinsdag 27 september 2011
Xbox Linux kernel development
As the development of Xbox Linux patches have halted since 2.6.16 (2006-03-28) I had the wild idea to develop the patches further to be compatible with the newer kernel sources (2.6.38 or 3).
The problem is that I don't have a lot of experience in C and none in assembly needless to say is that I haven't got any experience developing for kernels or drivers. I think this will be a nice challenge, good for the community (if anyone still uses the Xbox as a Linux machine) and it will broaden my knowledge of C and the Linux kernel.
The problem is that I don't have a lot of experience in C and none in assembly needless to say is that I haven't got any experience developing for kernels or drivers. I think this will be a nice challenge, good for the community (if anyone still uses the Xbox as a Linux machine) and it will broaden my knowledge of C and the Linux kernel.
Xbox server - W.I.P. #3
Friday I began connecting the VGA PCB to the Xbox, I finished this task Saturday so of course I tried the VGA PCB right away.
I then found out that I didn't got any image on the screen. First I thought this was because my monitor possibly wasn't compatible with the signals. So I tried a second monitor and again no image...
I didn't bother to troubleshoot until yesterday. Yesterday (of course when I wasn't at home) I thought of the mode setting that the Xbox requires and which wasn't handled by the PCB and possibly wasn't connected. This would explain why I couldn't get any image on both screens. So when I got home I saw the mode setting pins unconnected, this I immediately fixed in no time. I ran a test and.. again no image.
After a bit of searching I found out that to have a VGA compatible RGB signal the Xbox has to be set to NTSC in stead of PAL which is the default video mode here in Europe. Again this was fixed in no time. But still no image.
The mode setting was correct (mode 1+3) and the signal was NTSC so what else could be wrong?
The Xbox also needs a VGA compatible BIOS and I knew I had one. Cromwell to be exact. So I searched and searched and found nothing that resembled my problem.
So Cromwell didn't give me any visual response. Well I thought, let's try another BIOS, maybe it's just Cromwell trolling me. I still had a copy of Eurasia's generic flash disk laying around and a BIOS binary (Evox m8) on a CD so I flashed this. After flashing I rebooted the Xbox and kept my fingers crossed. And.. my monitor light changed from orange to green and I saw the flubber thing you're supposed to get. VGA worked!! A bit greenish but it worked.
Next I downloaded a VGA compatible BIOS, flashed it with Eurasia's flash disk and it did what I hoped it would. It corrected the colors and VGA was up and running. But how about Cromwell?
I again flashed Cromwell on the Xbox and rebooted the Xbox. Again no image. So maybe VGA isn't initialized at boot, but after Linux has taken control of the display? To check if my predictions were right I burned a copy of Xebian (as it was smaller to download/burn on a disk), popped it in the Xbox and booted it.
To my surprise it took a few seconds and then I saw a shiny happy penguin smiling at me. The display quality was better then expected, the colors were correct, I felt almost as satisfied as Tux.
Here's how I connected my VGA PCB.
The HSYNC and VSYNC points are accessible from the bottom side of the board. I took a bit of soldering tin and for both points I made a cone shaped pin.
Now it was time to solder wires to these pins. As always I stripped, twisted and plated the wires with tin. I then reheated the tin pins and connected the wires to the points.
Below is a picture of the Xbox main board where the AVIP connector used to be. I cleared the holes with very thin solder wick. After this the board was covered in remains of flux. I then cleaned the board with some acetone and a q-tip. The pin numbers are clearly exposed. and the board looks as good as new. The sync wires are visible on the right side of the picture.
The pin-out of the connector is as folowing:
The whole setup looked like this. In the top left is a composite to VGA adapter visible that provided me VGA output while my mod wasn't working, in the top right is the power supply visible, in the bottom left a 2.5" hard drive (the DVD drive is beneath the hard drive, there is a piece of paper in between to isolate the both), in the bottom right is the Xbox visible and in the middle is the PCB with a VGA connector connected.
And this was the display I did it for. As you can see the colors are normal and the display quality is like that of a normal VGA output.
I then found out that I didn't got any image on the screen. First I thought this was because my monitor possibly wasn't compatible with the signals. So I tried a second monitor and again no image...
I didn't bother to troubleshoot until yesterday. Yesterday (of course when I wasn't at home) I thought of the mode setting that the Xbox requires and which wasn't handled by the PCB and possibly wasn't connected. This would explain why I couldn't get any image on both screens. So when I got home I saw the mode setting pins unconnected, this I immediately fixed in no time. I ran a test and.. again no image.
After a bit of searching I found out that to have a VGA compatible RGB signal the Xbox has to be set to NTSC in stead of PAL which is the default video mode here in Europe. Again this was fixed in no time. But still no image.
The mode setting was correct (mode 1+3) and the signal was NTSC so what else could be wrong?
The Xbox also needs a VGA compatible BIOS and I knew I had one. Cromwell to be exact. So I searched and searched and found nothing that resembled my problem.
So Cromwell didn't give me any visual response. Well I thought, let's try another BIOS, maybe it's just Cromwell trolling me. I still had a copy of Eurasia's generic flash disk laying around and a BIOS binary (Evox m8) on a CD so I flashed this. After flashing I rebooted the Xbox and kept my fingers crossed. And.. my monitor light changed from orange to green and I saw the flubber thing you're supposed to get. VGA worked!! A bit greenish but it worked.
Next I downloaded a VGA compatible BIOS, flashed it with Eurasia's flash disk and it did what I hoped it would. It corrected the colors and VGA was up and running. But how about Cromwell?
I again flashed Cromwell on the Xbox and rebooted the Xbox. Again no image. So maybe VGA isn't initialized at boot, but after Linux has taken control of the display? To check if my predictions were right I burned a copy of Xebian (as it was smaller to download/burn on a disk), popped it in the Xbox and booted it.
To my surprise it took a few seconds and then I saw a shiny happy penguin smiling at me. The display quality was better then expected, the colors were correct, I felt almost as satisfied as Tux.
Here's how I connected my VGA PCB.
The HSYNC and VSYNC points are accessible from the bottom side of the board. I took a bit of soldering tin and for both points I made a cone shaped pin.
Now it was time to solder wires to these pins. As always I stripped, twisted and plated the wires with tin. I then reheated the tin pins and connected the wires to the points.
Below is a picture of the Xbox main board where the AVIP connector used to be. I cleared the holes with very thin solder wick. After this the board was covered in remains of flux. I then cleaned the board with some acetone and a q-tip. The pin numbers are clearly exposed. and the board looks as good as new. The sync wires are visible on the right side of the picture.
The pin-out of the connector is as folowing:
| 1 VCC | 13 Mode select 3 |
| 2 audio right | 14 Mode GND |
| 3 audio left | 15 +12v |
| 4 audio right GND | 16 GND |
| 5 audio left GND | 17 pin19 GND |
| 6 sp-dif digital audio | 18 var (Blue) |
| 7 H-Sync | 19 var (Red) |
| 8 V-Sync | 20 pin18 GND |
| 9 Mode select 1 | 21 pin23 GND |
| 10 Mode GND | 22 var (Green) |
| 11 Mode select 2 | 23 var (Composite) |
| 12 Mode GND | 24 pin22 GND |
The whole setup looked like this. In the top left is a composite to VGA adapter visible that provided me VGA output while my mod wasn't working, in the top right is the power supply visible, in the bottom left a 2.5" hard drive (the DVD drive is beneath the hard drive, there is a piece of paper in between to isolate the both), in the bottom right is the Xbox visible and in the middle is the PCB with a VGA connector connected.
And this was the display I did it for. As you can see the colors are normal and the display quality is like that of a normal VGA output.
Abonneren op:
Posts (Atom)