Installing DBFit for SQL Server

I’ve been using DBFit on several projects recently, and I’m really pleased with the results. If you’re not familiar with DBFit, it’s a set of Fit fixtures that allows Fitnesse tests to execute directly against a database, without you having to build a separate connector. This post shows you how to get started with DbFit with the least amount of pain.

We practice Test Driven Development at ext.IT and while TDD is a pretty mature discipline in most modern programming languages, it’s still not widely practiced in the database world. We’ve had some success in the past using TSQLUnit, but I prefer DbFit because you can use it to write Unit Tests as well as Acceptance Tests. In addition, Fitnesse allows you to add additional colour in the form of documentation to truly turn your acceptance tests into an executable specification.

One thing that doesn’t (at least to me) appear to be clear from the documentation is that to install DbFit to test a Microsoft SQL Server database, you can simply install Fitnesse, and then install FitSharp. The advantage of doing this, compared to installing DbFit from sourceforge is that you will be working with the latest and greatest versions of Fitnesse and FitSharp. The current instance of DbFit on sourceforge uses a Fitnesse build from 2008.

Here is a high level overview of the installation steps (more detailed information including troubleshooting steps is available at each of the links below):

1. If necessary, install Java 6 from here: http://www.java.com/en/download/index.jsp
2. Install Fitnesse from here: http://fitnesse.org/FrontPage.FitNesseDevelopment.DownLoad. As it mentions on the downloads page, just download fitnesse.jar, into the folder in which you’d like to install it  and type java -jar fitnesse.jar -p 8080 (where 8080 is the port number you’d like to use to run fitnesse from – usually port 80 is taken by other stuff on your machine) at the command line. Fitnesse will unpack and install itself.
3. If necessary, install .NET framework 3.5 or 4.0 from here: http://msdn.microsoft.com/en-us/netframework/aa569263
4. Install FitSharp from here (choose the correct version for your version of the .NET framework): https://github.com/jediwhale/fitsharp/downloads. To install FitSharp, create a folder under the folder into which you installed Fitnesse (I name mine FitSharp) and unpack the installation files into this folder.
5. Once you’re installed and up and running, follow the examples in the DbFit Reference to see how the whole thing works.
6. One additional tip is that if you’re following the Hello World example in the DbFit reference, use the following text instead of the one they document in Step 2: Setting Up the Environment. This will correctly point you to the location of your FitSharp installation files and use the Microsoft SQL Server flavour of DbFit:

!define COMMAND_PATTERN {%m -r fitnesse.fitserver.FitServer,"FitSharp\fit.dll" %p}
!define TEST_RUNNER {FitSharp\Runner.exe}
!define PATH_SEPARATOR {;}
!path FitSharp\dbfit.sqlserver.dll

I’ll be posting some further information on using DbFit as an executable specification that includes acceptance tests, for Unit Testing and performance testing, as well as showing you some of the ways we’ve organized our test suites to make them more maintainable and easier to understand. Happy Database Testing!

Launching Cast Iron Studio 5 on Windows 7

We were recently working on a customer’s Cast Iron projects that had been upgraded from version 4.5 to version 5.0 and ran into a problem launching Cast Iron Studio 5.0.0.6, which we solved.

I thought I’d throw this blog post out as I couldn’t find any help from Google about this problem, so if anyone else is running into this, then you will be able to find an answer without having to open a support ticket with IBM!

Problem

After installing Cast Iron Studio 5.0.0.6 on Windows 7, the Studio will launch correctly the first time, but each subsequent time you try to run Cast Iron Studio, it will show the splash screen then close the application.

If you look in the installation folder, you will see an error.txt file that contains something similar to this:

org.osgi.framework.BundleException: A fragment bundle cannot be started: update@plugins/com.approuter.module.jde.tp-1.0.0.jar [77]
at org.eclipse.osgi.framework.internal.core.BundleFragment.startWorker(BundleFragment.java:224)
at org.eclipse.osgi.framework.internal.core.AbstractBundle.start(AbstractBundle.java:265)
at org.eclipse.osgi.framework.internal.core.AbstractBundle.start(AbstractBundle.java:257)
at com.approuter.studio.plugins.BootstrapActivator.startApplication(BootstrapActivator.java:60)
at com.approuter.studio.plugins.BootstrapActivator.start(BootstrapActivator.java:39)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl$2.run(BundleContextImpl.java:1009)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.startActivator(BundleContextImpl.java:1003)
at org.eclipse.osgi.framework.internal.core.BundleContextImpl.start(BundleContextImpl.java:984)
at org.eclipse.osgi.framework.internal.core.BundleHost.startWorker(BundleHost.java:346)
at org.eclipse.osgi.framework.internal.core.AbstractBundle.resume(AbstractBundle.java:355)
at org.eclipse.osgi.framework.internal.core.Framework.resumeBundle(Framework.java:1074)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.resumeBundles(StartLevelManager.java:616)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.incFWSL(StartLevelManager.java:508)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.doSetStartLevel(StartLevelManager.java:299)
at org.eclipse.osgi.framework.internal.core.StartLevelManager.dispatchEvent(StartLevelManager.java:489)
at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:211)
at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:321)

Solution

The solution is simple – launch Cast Iron Studio with Administrator privileges. You can of course right click the application from the start menu short cut and choose Run as administrator, but to avoid having to right click every time, follow these steps:

1. Go to the desktop shortcut for Cast Iron Studio (if you didn’t create a shortcut when you installed Cast Iron, then simply right click the Cast Iron Studio start menu item and click Send To…, Desktop (create shortcut).
2. Right click the short cut and click Properties.
3. From the Shortcut tab, click the Advanced button.
4. Check the box that says Run as administrator.
5. Click OK to close the Advanced properties dialog and click OK to close the shortcut properties dialog.

If you’d prefer to use a start menu item, rather than a shortcut, simply right click the short cut and cilck Pin to Start Menu.

Now Cast Iron Studio will launch successfully.

Let me know if this blog post helped you, or if there is a simpler work around that exists; this was an odd problem that didn’t appear to have a well documented solution that Google could find for me.

Bob’s Report – The cost of redundant processes

Have you ever been in a situation where you wondered where the original business requirement came from?

“Bob’s report must be produced every Monday morning by 10am!”

1. Mary extracts the data from the database
2. Bill adds the data to a spreadsheet and applies some pivots to massage the data
3. Sue looks at the report and makes sure there are no errors
4. Harry applies some text to the report and finishes off the formatting, then emails the report to Bob
5. Bob looks through the 200 unread emails in his inbox and tries to prioritize them. In the end he glances at “his” report for 30 seconds and thinks nothing more of it.

Where did the requirement for Bob’s report come from? Lets look at the history behind this requirement.

1996 – A new application is implemented for tracking the sale of widgets

1997 – Jean the head of sales has the requirement for a sales report for her weekly meeting with the CEO at midday on a Monday. She describes her requirements to the business analyst implementing the application. The format is decided and the process to produce the report is documented and discussed with the parties involved

2005 – Jean leaves Widgets inc. for a new position at another company. Bob is employed as head of sales. The CEO and Bob decide that they are going to meet on a Wednesday every other week to discuss the sale of widgets.

“Bob’s report” (originally Jean’s report) hasn’t been reassessed since 1997. Since 1997 Widgets inc have changed personnel and business processes have been updated. Technology has changed since 1997 and the process of producing Bob’s report should be much easier with a more modern approach (automated processing with Microsoft SQL Server Reporting Services, Business Objects, Cognos and many more). What’s more, four people have spent hours of time on a Monday morning producing a report for the past 6 years and it is not even read!

Solving the problem

This may be an extreme example, but I can guarantee you will have come across a similar scenario either in the organization you work in or in other organizations if you are a consultant like me. In so many cases we hit a wall when we ask the question, “Why do we need X?” Many times the reason is political, but in some cases it is purely because of a historical requirement and no one is prepared to ask the question. As it turns out, Bob is unaware of the effort that goes into producing his report; if he knew he would stop this process or refine the contents of the report so that it is more useful.

As consultants or business analysts we are always likely to have to dig deeper until we can find the truth. One method that can be useful is to follow the “5 Whys“:

The 5 Whys is a questions-asking method used to explore the cause/effect relationships underlying a particular problem. Ultimately, the goal of applying the 5 Whys method is to determine a root cause of a defect or problem.

Quite often we find that the problem is several levels deep. Using the 5 Whys technique can help to get to the root cause of a business problem rather than the apparent problem on the surface, or even worse a problem that has not been identified.

As consultants our hardest task can be getting to talk to the right people. How can we reach our goal in a reasonable time without upsetting too many people? Here are some rules we can follow:

• Set expectations up front. You want X number of hours time from people in the relevant areas of the business. You want to be able to speak to them in a reasonable period of time.
• Speak to the right people. You are not likely to get a lot of the CEO’s time, but if his or her input is likely to help ask for their time. Be prepared with targeted questions where possible:
  • What reports do you currently receive? Do you use the data in these reports?
  • What do you use this data for?
  • What information is not made available to you that you would like to see?

• Don’t accept the answer “that’s just the way it’s done”. Use the 5 Whys to dig deeper until you find the real reason for a requirement.
• Stick to your guns!

Remember, getting down to the root cause will benefit the organization you are working for. They may be throwing thousands of dollars per year away on Bob’s reports.

Show them the numbers – it’s results that matter

On a recent project one of our customers had some questions about the project’s progress. The project was a mid-size systems integration between Oracle Financials and Onyx CRM for a Fortune 500 company.

As with many large companies, there were several business units and people with interest in the project, and also as with many large companies, people had several other projects to deal with and were not attending the daily standups or weekly progress meetings.

I began to hear that there were some concerns about the project progress from some of the management team and a concern that we were spending too much time writing tests instead of code.

This came as a big surprise to me: from the perspective of the team working on the project on a daily basis (both customer and ext.IT), we were sailing along with no problems. We were providing the usual artifacts such as status reports which showed how we were tracking against estimated hours and when our code complete date would be, and all of this was tracking well against our original estimates, so to hear that people were unhappy was a big shock.

I called a meeting with all of the stakeholders and as soon as I invited all of the stakeholders I realized two things:

1. This meeting would be the first time in the whole project that all of the stakeholders had been in a room together.
2. While we had discussed our approach during the initial engagement, we hadn’t held a project kickoff with all of the stakeholders to discuss our approach to building software.

Both of these items were my fault, but here I was, several months into the project with an unhappy customer, so I thought I would address it in this meeting.

I put together a short presentation which discussed our approach to building software, including brief descriptions of how we perform iterations, test driven development, automated acceptance testing, retrospectives and other benefits to the way we build software, and why we considered our approach to be best practice.

Almost as an afterthought, I ended the presentation with a couple of slides showing our project status. This was an afterthought because everyone at the meeting was a recipient of the weekly status report containing the same information. I created one slide comparing our original project estimates (before we had detailed requirements) to our detailed estimates (once requirements were fleshed out) and to our actual hours so far on the completed features, and another slide showing our defect count and comparing it to another of the customer’s internal projects that they had build without the benefit of automated tests.

Once the meeting started, it was clear that I didn’t need any of the slides about our approach. We spent a few minutes discussing the original estimates against the actual time taken for completed features: everyone was happy with the job we had done in estimating against the original high-level requirements. We also discussed how we were tracking on the hours left: again, everyone was satisfied with the results. Finally, we spent five minutes comparing the low defect count on our project (with a high level of test automation), compared to some of the customer’s other projects (without unit tests), and everyone was satisfied with this too.

There wasn’t a single question about our approach, and ultimately no-one in the higher echelons of the customer’s business cared about the minutiae of our approach, nor were they interested in why we thought our approach was best practice.

They simply wanted to see the results of our approach.

In retrospect, this might seem obvious, but the best way to convince anyone that your approach to doing something is practical and pragmatic, is to do it and measure it against other approaches. People don’t care about dogma, or why you think something should be the best way. Only by showing people the numbers and measuring the effectiveness of your approach can you truly show that your practices are effective.

To conclude: After this meeting, the project was delivered successfully without any more concerns from the customer. We were one of several projects integrating with Oracle and we were the only project to deliver on time and under budget, with a very happy customer!

Data Smells

There has been much discussion in recent years about code smells and using them to drive refactoring.  I have heard and read very little, however, about data smells.  Data smells are different from code smells and they drive a different process, but I believe that they are equally as important.

Data is like fresh food from the grocery store; much of what you buy has a “use or freeze by” date.  Data too has this expiration date and like the fresh food, if this date is ignored, there will be spoilage that results in unpleasant smells.  In data’s case, the “use by” date can be quite obvious as in the case of accounting data or more subtle like the useful lifespan of customer purchasing data.  The “freeze by” date gives an indication of when the data is no longer useful in daily operations but generally has value as historical information to be used in reporting and analysis.  The most insidious data is comprised of the fields, tables and sometimes entire data stores that have been rarely or never used.  The bad smells don’t reach us because this data or structure is not being actively used.

Where does this data originate?  Sometimes, it comes from the pet projects of long-departed managers or executives that ultimately didn’t serve a useful business function.  Others may come from projects that were dropped due to budget or other business constraints and left in a partially-complete state.  In my experience, the greatest contributors to stale data are the large, enterprise applications like Onyx CRM / Consona CRM or Oracle Financials that have been implemented in a traditional waterfall fashion.  SaaS applications such as Salesforce.com that allow well-intentioned but technically-challenged users to create entire data structures are a close second and gaining rapidly.

In waterfall implementations of large, enterprise-level applications, user groups often pile on requirements in an effort not to miss any possible use case, even those which should be considered to be marginal.  Because requirements are gathered upfront and then cast in stone, it’s easy to see why so much marginally-useful or even useless data points are created to satisfy the perceived business need.   How often do we review this data once it has been implemented?  Monthly, yearly, once a decade?  Unfortunately, the likely answer is: never.  We have servers to manage, upgrades, updates and patches to test, database code to write and refactor, documentation to maintain, meetings to attend, newer, shinier applications to prepare and the list goes on and on.  Meanwhile, columns that contain minimal or no data and tables with stale, useless data take up space and use server resources unnecessarily.

It is incumbent on data professionals, and I include DBAs, data developers and business analysts in this category, to find these issues and to offer solutions.

• Consider re-purposing custom and user-defined data columns to support newer business needs.
• Offer to archive (just in case) those old tables and to eliminate them from the production server.
• Refactor code to eliminate dead-ends and processes that are creating the offending rows.  I have seen several cases where there hundreds of thousands of rows of data that is still being created to support a process that no longer exists or is assigned to a user who is long-gone.

We offer CRM “tune-ups” for Onyx CRM, Consona CRM, Oracle Financials, Microsoft Dynamics CRM and Salesforce.com which would include data analysis of this kind.  Look for a blog post on this in the future.

In addition, I’ll address the “how-to” side of this issue in a future post.  In the meantime, take a look at your biggest data stores.  Do you have any columns that have never been populated, have not been populated in more than a year or have a population of less than 5% of the rows?  If so, these are candidates for the big cleanup ahead.