How did I not know about this?

“Mintty is a terminal emulator for Cygwin. It is based on code from PuTTY 0.60 by Simon Tatham and team.”

Previously I’ve been using PuTTYcyg which is a version of PuTTY with Cygwin support added. It’s good, but mintty seems a lot neater – rather than being a hacked version of PuTTY it’s a dedicated application derived from it. Also, it’s actually packaged in Cygwin which is a plus.

This could almost be an elegant replacement for the (truly abysmal) Windows command prompt, except that a complete replacement would need to be compatible with the vast array of existing console applications – and that’s a rather difficult task for technical raisins. The author has also developed a program which should solve part of the issue, and deserves investigation.

As reported by just about everybody, those litigious bastards are at it again. The BBC has a reasonable summary, but Gizmodo seems to have the best coverage so far, including the list of (20!) patents, the full filings, some links to Apple’s related issues with Nokia ((six of one and half a dozen of the other there really)) and Palm, and a brief response from HTC.

It seems fairly clear this is mostly about competition from Android, but get this:

“We can sit by and watch competitors steal our patented inventions, or we can do something about it. We’ve decided to do something about it,” said Steve Jobs, Apple’s CEO. “We think competition is healthy, but competitors should create their own original technology, not steal ours.”

Edit:

Let’s compare that to his opinions when it’s the other way around:

That man is so full of shit.

Update:

HTC finally responds – sadly with just the expected content-free press release. ‘We’re really good’, ‘We innovate loads’, ‘We don’t think this is fair’, etc.

GNU screen is an excellent way of keeping a persistent multi-terminal session for console based programs. It shines when connecting to a remote server – ssh in, run screen to create a session, then do whatever you want in it. At any point you can detach from the session and it will keep running in the background until you reattach it (or screen is killed for some reason; good luck maintaining a persistant session across reboots). This works even if your connection is reset by Pierre, or whatever.

An SSH authentication agent (such as ssh-agent or Pageant) is a similarly excellent way of securely ((As long as you trust root on any intermediate servers. For more on that look here, especially the section on ‘Agent Security Concerns’)) forwarding ssh authentication requests, allowing for something resembling single-sign-on between machines configured to accept your ssh key.

There is one problem however: the ssh client knows to connect to a running agent based on the environment variables the agent provides. On the other hand, screen inherits its environment from the shell that started it in the first place. When you reconnect, you are presented with exactly the environment you left, which has a different ssh-agent magic token – leaving ssh unable to find the agent.

Enter the following handy function:

function ssh-reagent () {
    for agent in /tmp/ssh-*/agent.*; do
        export SSH_AUTH_SOCK=$agent
        if ssh-add -l 2>&1 > /dev/null; then
            echo Found working SSH Agent:
            ssh-add -l
            return
        fi
    done
    . ~/.ssh-agent
    if ssh-add -l 2>&1 > /dev/null; then
        echo Found working SSH Agent:
        ssh-add -l
        return
    fi
    echo Cannot find ssh agent - maybe you should reconnect and forward it?
}

Which I got from this guy. Put this in your ~/.bashrc ((Or the equivalent for your shell of choice, if it acceps the same syntax)) and then you can type ‘ssh-reagent’ to set the appropriate environment variable. Alternatively, if you prefer a greater degree of automation, the author has a further development.

NB: My version has two differences to the original version in that post The first is a bugfix changing 2&>1 to 2>&1; see here if you don’t understand that. The second is to source ~/.ssh-agent; unfortunately I can’t remember doing that and I don’t seem to have a decent commit log for it, so I’ll be damned if I can remember what problem it fixed :P.

For some reason (product differentiation? simple oversight?) VMWare Server 2 ((Downloadable from https://www.vmware.com/tryvmware/)) has no option to create a VM using a physical disk. Fortunately, that doesn’t mean that it can’t use them; you just need to import a predefined VM and tell it to use that. It’s possible to create a VM in the right format using some other version of VMWare – Workstation, an older version of Server, and possibly Fusion – but that means you need to have those around.

Alternatively, there are some third-party tools to do the job. Take a look at VMX Builder; it’s a free (as in beer) Windows program, that’s said to work in Wine – though if you’re running Linux you’d probably be better with KVM and virt-manager than VMWare. VMX Builder makes it straightforward to define a VM as you need (NB: if you have a 64-bit guest you may need to choose the LSI Logic virtual SCSI controller).

Having done that, go to the VMWare console and choose ‘Add Virtual Machine to Inventory’ from the ‘Virtual Machine’ menu.

Using this method it’s possible to boot a real physical Linux installation from within Windows and have it run in the background as a service ((If you’re doing that it’s probably worth looking at open-vm-tools which is probably packaged by your distribution of choice, and probably more likely to actually work than VMWare’s official version.)). Most virtualisation options on Windows are either a) expensive, b) arbitrarily limited – 32-bit only, single-processor only, etc. or c) intended to run in the foreground as a standard windowed application.

Update: Newer versions of amavisd-new (certainly 2.6.4-3) have a variable called $logline_maxlen which does exactly what it says on the tin. The rest of this post is therefore of only historic interest.

I administer a mail server that uses the popular amavisd-new to perform virus scanning, and spam filtering using SpamAssassin. That server uses Logcheck to send mail notifications if any unusual messages are logged – it works by filtering out any log messages fitting a set of patterns considered unimportant, and reporting any that are left. One small annoyance with amavis is that it splits lines that are longer than a certain magic number, so you get syslog entries like this:

Feb 23 15:15:23 hostname amavis[3139]: (03139-11) Imagine this message is long en....
Feb 23 15:15:23 hostname amavis[3139]: (03139-11) ...ough to be split

The split means that neither line fits the pattern of a message which can be ignored. Since you can’t predict how long the message will be, it’s not possible to write an ignore rule to take this splitting into account.

Extensive Googling revealed no mention of this problem anywhere, and eventually I decided I would have to Use The Source. The business part of amavis is the script /usr/sbin/amavisd-new – at least that’s where the Debian package puts it – which includes the following section ((as of version 1:2.6.1.dfsg-1)):

1876
1877
1878
1879
1880
1881
1882
1883
1884
1885

    my($pre) = $alert_mark;
    my($logline_size) = 980;  # less than  (1023 - prefix)
    while (length($am_id)+length($pre)+length($errmsg) > $logline_size) {
      my($avail) = $logline_size - length($am_id . $pre . "...");
      $log_lines++; $! = 0;
      syslog($prio, "%s", $am_id . $pre . substr($errmsg,0,$avail) . "...");
      if ($! != 0) { $log_warnings++; $log_status_counts{"$!"}++ }
      $pre = $alert_mark . "...";  $errmsg = substr($errmsg, $avail);
    }
    $log_lines++; $! = 0; syslog($prio, "%s", $am_id . $pre . $errmsg);

As we can see, there’s a hardcoded constant which determines how long a line may be. Removing or commenting out the undesired section leaves us with

1876
1877

    my($pre) = $alert_mark;
    $log_lines++; $! = 0; syslog($prio, "%s", $am_id . $pre . $errmsg);

Another option might be to increase the value of $logline_size to something that means you rarely if ever exceed it.

What could possibly go wrong?

Generally it’s a very bad idea to start messing with files that are managed by the distribution’s packaging system – for one thing, it’ll be overwritten the next time there’s an update. Additionally, there must presumably have been a reason for the line splitting in the first place, and hence a reason not to disable it. This is a dirty hack, and I don’t recommend doing it. I’m only even posting it so that in the future there might be some record that the issue exists.

That said, so far it’s working as desired and there have been no side effects.

Some Background:

BTW: In the context of this post, ‘Windows’ always means ‘Windows NT’.

The Windows [[Kernel_(Computing)|kernel]] is designed to support multiple independent subsystems – different application environments – atop its [[Native_API|native API]]. The two most common examples of Windows subsystems are the [[Windows_API|Windows API]] – formerly known as Win32 and used by the overwhelming majority of Windows programs, and the [[POSIX]] subsystem ((Supposedly created because POSIX compliance was a checklist requirement for US government software purchases)) provided by [[Microsoft_Windows_Services_for_UNIX|SFU/SUA/Interix]] ((Originally Windows came with a [[Microsoft_POSIX_subsystem|basic POSIX subsystem]] out of the box, but that died with the release of Windows XP, replaced by Interix))- henceforth known as Interix.

Note: Just to muddy the waters, Cygwin provides a POSIX environment layered on top of the Windows API.
This has nothing to do with the POSIX subsystem.

The Problem

The Windows API is – by default – case preserving, but not case sensitive, everywhere. On the other hand, [[NTFS]] (the standard Windows filesystem) is POSIX compliant – presumably because the POSIX subsystem would have been unable to claim compliance otherwise. This means among other things that it supports case sensitivity – with the result that it’s possible to create two files which differ only in case, mightily confusing most Windows software. Even worse, POSIX allows file names to contain characters that the Windows API does not. Now you don’t need two similarly named files to create confusion; just having a single file named, say, ‘How Soon is Now?.flac’ will cause a problem.

So What?

Normally this wouldn’t be an issue. Coming from Unix it seems pretty annoying, but livable. A problem arises however when a filesystem is shared between Windows and an OS which will happily create files using the full range of names allowed by POSIX (in practice this means most operating systems other than Windows) .

Say some unsuspecting user (cough) is using [[NTFS-3G]] to access NTFS filesystems from Linux; by default it doesn’t restrict the filenames allowed to the Windows-safe subset.

Well, I have ((Or rather ‘had’ – it’s all been renamed for Windows-safety now)) a lot of music filed away such that the file is named after the title of the track, and a handful of tracks include characters which are verboten under Windows – most commonly a colon or a question mark. It took quite a while to figure out why a lot of my music seemed to be mysteriously inaccessible from Windows – and wouldn’t it be nice if there were some way to get to them, even if just to rename them? Well apparently “all the files are accessible if SFU is installed on Windows”. There’s only one problem with that: Interix/SFU isn’t available for [[Windows_XP_Professional_x64_Edition|the version of Windows I’m using]]. Sigh

What Next?

Clearly Windows itself is capable of dealing with the full range of filenames, otherwise Interix wouldn’t be able to use them. Is there any way for the Windows API to access this capability? Investigations reveal the existance of a promising-sounding flag to CreateFile: FILE_FLAG_POSIX_SEMANTICS. This flag’s description doesn’t agree with its name – the description only talks about case sensitivity. Is that an oversight in the description, or a poor choice of name? Further investigations require a small detour…

At some point Microsoft decided that the ability to have files whose names differ only in case could be dangerously confusing for some of the extant Windows software (think antiviruses trying to scan MALWARE.dll vs. malware.dll), so case sensitivity is disabled by default across the board. It can be enabled for subsystems other than the Windows API, and for the Windows API using FILE_FLAG_POSIX_SEMANTICS, by setting a registry key:

Under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\kernel, there needs to be a DWORD called ObCaseInsensitive, with the value ‘0’.

…having set that key and rebooted, the FILE_FLAG_POSIX_SEMANTICS flag should now have an effect. A quick Python test confirms this:

f1=win32file.CreateFile('foo', win32file.GENERIC_WRITE, 0, None, win32file.CREATE_NEW, win32file.FILE_FLAG_POSIX_SEMANTICS, None)
f2=win32file.CreateFile('FOO', win32file.GENERIC_WRITE, 0, None, win32file.CREATE_NEW, win32file.FILE_FLAG_POSIX_SEMANTICS, None)

Both files are created successfully, and can be seen in Windows Explorer, though it can’t differentiate between them.

So, now we get to find out whether that flag really does allow POSIX semantics as its name suggests, or if the description is correct and all this was for nothing:

f3=win32file.CreateFile('foo?', win32file.GENERIC_WRITE, 0, None, win32file.CREATE_NEW, win32file.FILE_FLAG_POSIX_SEMANTICS, None)

Result: error: (123, 'CreateFile', 'The filename, directory name, or volume label syntax is incorrect.')

Damn.

Update:

Cygwin 1.7 claims to support case sensitivity with the appropriate registry key set, and also the full range of characters in filenames. Since Cygwin does use the Native API for some of its operations, I dared to hope that perhaps they’d performed some black magic to get this to work properly. Sadly, attempting to read an existing file with a question mark in the name leads to the response ‘cannot access <filename>: No such file or directory‘, so I guess they’re just faking it.