TiddlyWikis and a web server

There are a number of options around for making tiddlywikis editable across the web. However, what if all you want to do is to be able to edit locally and have it viewed publicly? If you want to develop on a laptop and make notes along the way, then you would need to edit the file and copy it to the web server across the network. Remembering to do that is just too much work.
Even if you were happy to edit the file directly on the web server across the network, there is a problem in that the tiddlywiki will fail on save if you are accessing it from a network drive.

I tried synchronizing folders with the web server but ran into the problem of saving the wiki, it must have something to do with the naming of the synchronization directory.

But there is a really simple answer, and it integrates into something you are probably already doing. Robocopy! I already have scheduled robocopy scripts that are backing up files to an external drive on the network. Well then, just increase the script to take your wikis to your web server ... done!

The great thing about this approach is that your blog can link to your wikis. Wikis can be more domain oriented, and within the wiki you have your tags for crosslinking.

Who are you impersonating?

Been going through the pains of installing VS Team Foundation Server.
Here are a couple of pointers for others who may be suffering through the same process.
Make sure that Sharepoint, Reporting Services and TFS are all using the same account. That is, LOCAL SERVICE or NETWORK SERVICE but not a combination of BOTH.
Have the user that is creating a process also be a content manager and a administrator within Reporting Services.

Communicating with legacy Source Code Control

Had a situation where I needed an old source control system written in Foxpro to pick up on file changes and consolidate them. The file changes could occur via any external package, but I was only interested in files in certain directories.

There were a couple of ways to handle the file once the change had been detected, but how to detect the change? The answer came by .Net's fileSystemWatcher. Run a .Net service that watches for changes. The next step was then how to communicate the changes to VFP, the answer there was sockets. I could have used interop, but the nice thing about sockets is that they are light.

The result was a simple event handler that detects a change and sends a message to the Source Code Control that it has something to process :

/// 

/// Track changes in the filesystem and pass them over to the legacy SCC system.
/// 
/// 
/// 
protected void writeFileEvent(object sender, FileSystemEventArgs e)
{
 string cModFile = e.FullPath.ToString();
 cModFile = cModFile.ToLower();
 // Exclude files that don't have to be stored
 if (cModFile.IndexOf("develop\\pending") != -1 ||
      Path.GetExtension(cModFile) == "" ||
      cModFile.IndexOf("develop\\bin") != -1 ||
      cModFile.IndexOf("\\prog\\ide.log") != -1 ||
      cModFile.IndexOf("\\prog\\symbols\\") != -1 ||
      cModFile.IndexOf("\\prog") == -1){
  return;
 }
 // Record the file to be processed
 string cLogFile = "\\formwork\\develop\\pending\\" + 
            Guid.NewGuid().ToString() +".xml";
        // Create an instance of StreamWriter to write text to a file.
        // The using statement also closes the StreamWriter.
        using (StreamWriter sw = new StreamWriter(cLogFile)) 
        {
            // Add some text to the file.
            sw.Write("" +
             "" +
             cModFile + "");
            sw.Close();
        }
  
        //create a new client socket ...
 // Don't really have to wait for a response. 
        // The file has been recorded (above) for processing
 // so the next time the SCC starts it will pick up on the change anyway.
 // However, if the SCC is running now let it know there is work to be done so
 // it doesn't have to poll the directory.
        Socket m_socClient = new Socket(AddressFamily.InterNetwork,
                   SocketType.Stream, ProtocolType.Tcp);
        String szIPSelected  = "127.0.0.1";
        int  alPort = 8000;
        
        System.Net.IPAddress remoteIPAddress = 
                    System.Net.IPAddress.Parse(szIPSelected);
        System.Net.IPEndPoint remoteEndPoint = 
                    new System.Net.IPEndPoint(remoteIPAddress, alPort);
        m_socClient.Connect(remoteEndPoint);
        String szData = cLogFile;
        byte[] byData = System.Text.Encoding.ASCII.GetBytes(szData);
        m_socClient.Send(byData);
 m_socClient.Close ();
}

Define your programming skills via the collective conciousness

OK, I know that this might not be what Jung had in mind. But let's look at programming language trends through a global lens. What language should you invest time in? What languages should your business be investing in. The problem is, of course, that when you pick a language / operating system you commit to a platform for many many years. So what is best practice? What are other people doing?
Well the solution is to look at what jobs are being offered. indeed.com has a great interface for doing just that. The 3 that I want to look at are Python, C# and Java. Now there might be lots more but I want security:

• The software I write needs to be maintainable both now and in the future. So I need to be able to find people with the skills I need. Therefore, the trend has to be strong.
• The skills I have must be marketable in the future, because it is a serious investment in time to update your programming skills.

Python is somewhat of a risk in that there are not currently that many Python jobs around but the trend is definitely there and C# is a given. Java just does not seem to have the same momentum.

python Job Trends

python jobs

c# Job Trends

c# jobs

java Job Trends

java jobs

The 3 issues of cloud computing

3 issues of cloud computing

One of the problems of cloud computing is how safe your client's feel knowing that you can view their data if it is loaded onto your servers. There is nothing I can do about this viewing problem, except to say that even if a client has the application loaded locally their feelings of security are largely misplaced. Most programmers would know how to transfer every bit of that data to some remote site. The relationship of client to application supplier is essentially one of trust.

Another problem with cloud computing that if your company name isn't Amazon or Google, well nobody is going to feel safe having all their data in your cloud. What if your company disappears? One final issue with cloud computing is that if a client needs to interface their data with other in-house systems.
It turns out these last 2 issues can be solved in 1 fell swoop. Data replication.

What is required is that the client runs a process that polls the server, and downloads any changes to a local database. One key to this solution is that every record on the server needs a modification timestamp. Then it is relatively easy for the local polling service to request all changes since ????. This is how the problem at http://www.freightfreedom.com/ was solved. The last pieces to the puzzle was to make it cross-platform, and to abstract the database layer so that the client could choose the database to store the local data into.

So the solution is data replication, 2 out of 3 ain't bad.

Building Systems

Been having a good look at Government 2.0 which I think is great. Particularly, http://www.katelundy.com.au/category/campaigns/publicsphere/open-gov/, the Public Sphere 2.0 is a great forum and hopefully we will see more of this public engagement in the future.

The problem is that whilst promoting social networking to derive policy is great, I think a fundamental issue is still to be resolved .. the systems behind the scenes. I am not disparaging the process at all, but to me it is a tune I have heard before. The users want everything and the systems behind the scenes are a tangled web of confusion. I believe you have to get the foundations right and that starts with how software is designed, how it is built and how it processes data.

So here are my guidelines for software development just in case someone stumbles on them. It literally would save millions of dollars if government IT systems were built with a few prerequisites in place prior to any project commencing. There are many guidelines available for managing IT projects, I will preface this by saying I don't know what the government already has in place.

As IT integrates more and more with our everyday lives it is imperative that the government;

• Not be tied to proprietary systems
• To build systems that will have significant life spans
• build systems that can communicate well together.

So from an ignorant view the following would have to be a bare minimum;
1. UML. UML provides a common graphical language for exchanging system design. This should be part of the visualization of government processes. While the UML specification has many diagrams in it the 'Use cases' would allow lay people the ability to visualize a system flow without being inundated with technical jargon.
2. Code isolation. UML and unit tests encourage applications to be broken into small code blocks. These code blocks can be run under unit tests isolated from the production environment and suits the agile development philosophy.
3. Unit tests. Unit tests, allow new versions of software to be tested, as an isolated module, prior to implementation minimizing the possibility of failure. These are written prior to the software code being developed. Using unit tests and contract programming would allow the outsourcing of non-critical routines, which would vastly decrease the cost of developing and maintaining systems.
4. Language choice. Proprietary languages are a risky investment. I worked for the X government department as a contractor when the Ochre language went bankrupt. At least 3 years of work was effectively lost. The 2 contenders for ubiquitous programming are Python and C#. Both languages are now available on Linux and Windows, both allow database abstraction. Productivity in Python is higher and it is closer to the human readable form. C# is excellent in multithreading and operating system integration. Python can be integrated with C# and C++ to get the best of the lower level languages. Both Microsoft and Google are using it.
5. Database abstraction. This is crucial. Code should not be written in such a way that it talks to a database directly. It should go through a database abstraction layer first. This means that the application is independent from the database platform. So should a system be implemented using MS SQL, it could be changed to run on PostgreSQL, Oracle or any other supported platform without any changes. Database Abstraction Layers are available in Python (SQLalchemy) and C# (nHibernate) as well as others. As part of database abstraction, application rules should not be stored in the database, they should reside in the application and work on the abstraction layer. If you run stored procedures you have effectively tied yourself to one platform.
6. Code repositories. Code should reside in a system that supports check-in and checkout, prior to an application being built. That ensures that the code cannot be lost.

The open source minefield

Needed to write a GUI, nothing fancy just some data binding. Tkinter is included within Python, great! Spend a bunch of time learning tkinter and its quirks, then I am done. Now to package my work of art with py2exe to deploy under windows .... splat.

Posted messages to the py2exe mailing list and waited, and waited. I suppose I could work it out, put in some hurclian effort and figure it all out. Trouble is I just don't have that much time, and the interface wasn't that complicated. Answer, lick some wounds, rewrite the interface in HTML and write a simple web server based on one of the activestate recipes.

Debugging Python threads

My understanding is that because pythonwin runs in its own thread if you spawn a thread - you can't debug it.
I have an application that runs 2 background threads. You can make special classes for testing the application outside of background threading, but that just gets confusing. I really don't want to be creating a whole bunch of code just to be able to debug what I already have.

I was really getting frustrated with the whole process, until I discovered winpdb. It is bloody mavellous, it really needs to be bundled into the activestate distribution.