0 comments

Refactoring pytddmon

I’ve been reading quite a lot about ”Domain Driven Design lately. It’s inspired me to think about how to re-design the 700 lines long ”blob” that is pytddmon.py into something more manageable and grokkable pieces. Hopefully even less lines!


AstridWestvang / Foter / CC BY-NC-ND

The basic idea is to use asynchronous messages to communicate between the different parts of the ”system”, which would be split up into ”actors” running in their own thread:

  1. A ”Heartbeat” thread/actor simply publishes ”beats” to the message bus at regular intervals
  2. A ”FileChangeMonitor” thread/actor subscribes to the beats, and checks for file system changes. If a change is detected, it publishes a ”FileChange” message.
  3. A ”TestRunner” thread/actor subscribes to the ”FileChange” message, and does the actual test running (using a separate Python interpreter process as usual – we still need that to avoid the import cache). When the run is over, it publishes a ”TestResult” message.
  4. A ”GUI” thread/actor subscibes to the TestResult message, and updates the appropriate UI widgets.

A nice side effect of this refactoring, apart from a humble hope that it will be simpler to unit test each part in isolation, is that extending the system becomes easier.

For example:

  • Exchanging the TkUI based actor with a console/text UI becomes almost trivial; just instantiate another class on pytddmon boot. The implementation of a TextUI is in itself trivial. Doing a text UI in the current architecture would mean duplication and a lot of headache!
  • Creating a plugin system, e.g. to make a plugin for snapshotting the source code after each test run, is almost trivial: just allow dynamic importing/instantiating of plugin modules, that can subscribe to the events published by pytddmon.

Of course, this needs some thought. For example, I still want to keep both the ”xcopy” installation model of pytddmon, aswell as trying to separate things into more files to ease development (700+ lines of code is not my idea of well-structured code…). This in turn puts more demands on the CI system (Travis) which needs to not only test one pytddmon.py file, but also ”assemble” that pytddmon.py file, aswell as test the separate-module-version of pytddmon.

We will also need a mechanism to actually send asynchronous messages, but after having read and tested Python threading over the summer, I think Queues and threads are all that’s needed for a straight forward implementation. Going in the ”as few dependencies as possible” philosophy of pytddmon I’ll likely roll my own class ”MessageBus” (which is the likely and conventional name for such a mechanism). Read more about the architecture over at wikipedias article on the Publish-Subscribe pattern.

 

Version 1.0.2 is now available for installation through the cheese shop:

pip install pytddmon

This version features a new command line argument, –no-pulse, which disables the ”heartbeat” of pytddmon:

python pytddmon.py --no-pulse
The Nick Page / Foter.com / CC BY

The background and idea with the pulse was twofold:

1) Give the impression of something being ”alive”, like a heart beat monitor in a hospital tells you the heart of the patient is working by giving a signal, pytddmon tells you your project is continously being tested.

2) Give an indication of when the file change scans where being performed, that is purely as a debugging tool.

 

pytddmon is now available in the Python Package Index (PyPI)

This means you can conveniently install it by issuing the following command (of course, assuming you have pip installed!):

pip install pytddmon

This is a contribution by Maximilien Riehl, thanks a lot!

 

0 comments

pytddmon on Ohloh

Thanks to Wari Wahab, pytddmon is now registered on Ohloh.

2 comments

git and github

pytddmon has started a move to github. That includes source repository, issue tracker and wiki.

 

In my previous post, I mentioned a serious bug that had kept my motivation low since autumn 2011.

The bug is about renaming .py files. Imagine you have a unit test test_some.py. It is testing some.py, and when pytddmon is running, all tests are green. You decide you want to rename some.py to something.py because of reason(tm). Since test_some.py is doing import some, but some.py doesn’t exist anymore, there will be an exception when trying to run the tests in test_some.py – which in turn fails that unit test. That should make pytddmon sad and red, right?

Wrong! It does not – pyddmon stays green*!

That is why I call it the rename bug (link to issue page).

I found it extremely odd for a long time, and I had a clear memory of this working at some point of pytddmons history. So I tested for the existence of this bug in every single revision (>100!) of pytddmon going back all the way to the launchpad days. Not in a single version did pytddmon work as expected; the bug was there all the time.

So I gave up for awhile. And then later on I built the Monitor class and refactored, refactored, refactored. Still not red!!! What the hell was going on?

Then it hit me how stupid I am. It was the .pyc files! They are left beside the .py file and CPython/pytddmon loads them instead of the renamed file! Face palm!

Anyway, to make a long story short, this sort of thing clearly is something out-of-scope for pytddmon to care for. For example, the exact same (mis)behavior exists when using basic unittest running: python test_some.py will import some.pyc and not something.py. However, in Python3.2 the rename problem is non-existent: .pyc files behave differently. Read PEP3147 for details (thanks for hint on this Neppord!).

One more reason to start using Python3.2 more! :)

*this is not true if you are using Python3.2. Read the rest of the blog post to understand why!

Long time no see!

Pytddmon has been asleep for awhile. The plans for configurability from the autumn didn’t fly, not least because I found out about a serious bug which made me unmotivated to merge stuff into the objarni/pytddmon repository.

But beneath the surface some things has started to happen. In an attempt to understand the bug, I tried to simplify pytddmon.py as much as possible. That work resulted in a new class for file change detection called Monitor. The work to integrate that class is going on in the pytddmon_monitor_integration fork, and so far it has resulted in 30% less code — down from 800 lines to 550! Even though it actually didn’t solve the bug :( , I’m all for keeping the source small and neat, so I think I will merge that fork as soon as it feels stable enough, and make a new release at the same time.

But the bug remains unsolved. However, I have understood the nature of the bug, and actually it might not be possible to solve at all, or rather, not a problem which pytddmon should solve for the user. I’ll write a blog post about it some other time.

Since we’re so few developers, we’ve decided to drop support for Python2.6, and focus on 2.7 (keeping CPython3.2 running too). If this bugs you out, please make your voice heard either by commenting here, telling the pytddmon twitter account about it, or join the mailing list and tell us about it.

Also, experimental work is being done to make pytddmon run in IronPython2.7 and Jython2.7. That would be seriously cool, seeing pytddmon in a .NET and Java ”suit”!

Oh yeah. Spam. I’ve had 500 spam comments since october. That is *way* too much administration, so I bit the sour apple and bought an Akismet key. So if you’ve added comments to this site and I haven’t answered, sorry. That problem doesn’t exist anymore.

I’ve added a page called ”How does it work in the menu to the right, explaining the essentials of how pytddmon functions.

I’ve put together a road map for pytddmon development efforts until christmas 2011.

October

We’re finishing a major code refactoring effort, using the pylint tool to increase code readibility. We’re already up from 6/10 to 9.5/10, so we’re almost finished with this.

November

Discussions about the next upcoming major feature: config file. Basically we want to make changing the behaviour of test-discovery and the monitoring functionality a bit more user-configurable than changing the source code of pytddmon.

December

Implement config-file feature.

So what’s happened since June?

The summer passed without much development, but in August things started happening again. Samuel (and I to some extent) have implemented Recursive Test Discovery, but it is not tested enough (especially not on Python3) to be released just yet. That feature is even more important than the config file feature (IMHO), so we’ll release that as soon as it’s stable. Another small change that’s been happening behind the scenes is the file name change from ”pyTDDmon.pyw” to the more conventional ”pytddmon.py”.

0 comments

doctest support

Neppord has contributed a couple of patches the last few weeks. Apart from some code-cleaning (thanks!) and a bug-fix (even bigger thanks!) he has added doctest-support to pytddmon. Read more on the details of the cool standard Python module doctest here.

Since doctest-tests are ”embedded” in the definition of a function (or class) in Python, the natural thing to do would be to scan all *.py files in the current directory. However, after some discussion, we’ve decided not do to that, based on the rationale that importing (and thus executing) every *.py file in the current directory may lead to unwanted* side effects.

Here’s some code to explain this:

def double(x):
    '''
    >>> double(5)
    10
    '''
    return x*2

If that function is saved in ”test_some.py”, pytddmon will discoverd it automatically, like any unittest-based test.

However, the natural thing is to save it in a code file, not a test file. If you save it to ”unit.py”, pytddmon will not discover it. Instead, you will have to specify that you really want pytddmon to look into that file for tests:

python pyTDDmon.pyw unit.py

The new version can be downloaded via the Download link to the right. It includes Python3 support too!

* Even though the Python convention is to not do things when a module is imported (as opposed to being run as a script) via the following well-known idiom:

if __name__ == '__main__':
    main()

… we consider that knowledge a little bit too non-trivial, and thus decided against letting pytddmon implicitly run all *.py-code automatically. Of course, test_*.py files are still run implicitly, just as has always been the case for pytddmon.

Sharing Buttons by Linksku