All → Software → FileSieve

A couple of new categories for the site!

Today someone actually registered on the Tracker.BootBlock.co.uk site for the first time, and as a bonus, they even created a nicely detailed bug report.

So I began working on fixing the bug in FileSieve (specifically, a plugin) and instead got side-tracked and added a preset set of filters for removing invalid OS characters from filenames.

Next up will be to add low-level support for removing/replacing invalid characters rather than shoving it into a plugin, and fixing the reported bug.

Edit: The date grouping issue on this site doesn't look like it has been fully resolved.

All → Software → FileSieve

Invalid characters in filenames suck.

Any plugins in FileSieve that return an invalid character results in the file the plugin is working on being skipped due to a failure when combining paths together. This is a controlled failure, I suppose you could say, but it's a failure nonetheless.

Despite adding an Invalid Filename Editor to the previous version of FileSieve, I disabled it as I wasn't sure at which step invalid characters should be substituted. Well, due to a quirk/bug involving filenames as reported by a user, I had another go at getting this working again.

Substitution now happens before path combining occurs, which I'm not 100% sure is the correct location. More of a semantic thing, but I'd like to move it to another step - not easy to do as this occurs during a (project-wide) plugin that does not reference the main, parent application.

All → Software → FileSieve

Finally got to the bottom of the issue that user was having with directory names in FileSieve.

Turns out it was due to a Modifier plugin affecting the entire path of items rather than just the path after the root destination.

FileSieve creates directories within a destination the user specifies, but the Find Replace modifier plugin was performing its modifications on the entire path, including the drive letter. There's absolutely no point to this (it was an oversight, originally - now corrected) as the destination already exists and cannot be modified (eg. changing case).

This user was using the Find Replace modifier in a really good way - replacing invalid characters with valid ones so FileSieve doesn't skip out on them. But using the previous (incorrect) entire-path-is-affected replacement style replaced the : colon character (as part of the drive letter) with a suitable replacement as specified by the user.

Oops. Now fixed; hopefully verified by the user himself as I've done a test build. A full release will be made shortly after verification.

While I was working with FileSieve anyway I also fixed some other stuff and re-instated the Invalid Filename Editor to replace any invalid characters encountered within filenames - a more consistent and low-level version of the Find Replace substitution method above.

All → Software → FileSieve

What a frantic couple of hours.

A user - the same as the previous post(s) - reported another couple of issues involving FileSieve, audio tag data, and Windows path and file names.

The first being that any audio files with a genre with a slash character (which is invalid in a Windows path - when not used a path separator - or filename) wasn't having those slashes replaced with a valid delimiter. This was an easy fix as the slashes were correctly being replaced, but any files that had multiple genres weren't having a delimiter appended after each genre (with the last dangling delimiter being stripped off the end), so you'd end up with a genre of FolkPop as opposed to Folk, Pop.

The second issue involves the way Windows treats directory - and probably file - names that end in a full-stop or a full-stop and a space. To put it bluntly: it doesn't bloody well like it.

FileSieve creates a directory based on a file's audio tag data, such as (minus quotes) D.I.S.C.O. but Windows internally, somewhere, "corrects" that to D.I.S.C.O thus stripping off the trailing full-stop. If there was a space after the full-stop, then it removes that, too.

Of course FileSieve doesn't see this ninja-change that happened internally within Windows and then chucks an error into the log saying that it couldn't find the file. That's because Windows stuck its D in our A.

Then I ran into an issue while beginning to debug the above path issue. Of course I did.

The (audio) Tag and (photo) EXIF plugins. They allow a mask to be set, such as %ALBUM_TITLE%\%TRACK%. %TITLE%.%FILE_EXTENSION%. When an (audio) file is processed, it would be placed into a directory and named as follows:

My Album\01. My Song.mp3

Except it doesn't. The file isn't actually renamed at all; the directory structure is created, but the filename is kept the same as the original.

This, as initial impressions appear, may be due to a core change made within FileSieve's processing. Plugins set a processed item's Path property independently of its Filename property. This means that plugins don't need a pile of additional logic to determine and set an item's path and filename. FileSieve's core will do that.

Unfortunately, it appears the Tag and EXIF plugins weren't updated to make use of this change and so treat both the path and file names as one property. Tomorrow, this changes.

All → Software → FileSieve

Oh maaaaan.

There's a gigantic issue with FileSieve and its plugins' settings. The wrong settings are being activated for the wrong Methods/Modifiers.

And it's an absolute nightmare to work out why, or where, or anything.

Another go tomorrow, then.

All → Software → FileSieve

Days and days to even work out why plugin settings weren't being set correctly and getting no-where, then this morning, within a couple of minutes of loading the project... it's fixed!

The issue was when a Profile was set, the Profile's active Method ("a method of sorting files") wasn't being initialised (which, among other things, passes the relevant settings on to the plugin). This doesn't make much sense as that's something that must have been happening previously otherwise it wouldn't work in the past. Was it broken all this time? Nineteen releases and no-one's mentioned it? Even I didn't realise?

That can't be right.

But, it's fixed, and I've created a new build and sent it to the user mentioned in previous posts to see if his original issues are fixed. If they are, then it'll be time to release 4.20 - a release all FileSieve users should update to.

Wondering if I should take a little break until I get the go ahead or if I should move this development blog over to the main domain...

All → Software → FileSieve

After the last Wake On LAN Ex release, FileSieve needs to be sorted out and released itself. This particular 4.20 release contains significant bug fixes that really needs to be put in the hands of FS users.

But when - just as before - I was just about to get the remaining issues fixed, I ran into another issue. A gigantic, showstopping issue that I'm absolutely confounded by.

Any time FileSieve does any (de)serialisation, JSON.NET throws exceptions; specifically, the .NET runtime throws exceptions. FileSieve worked absolutely fine forever - until this started happening yesterday. All of my other software that also (de)serialise are doing the same thing.

The code in-question (VB.NET):

Return JsonConvert.DeserializeObject(Of T)(tr.ReadToEnd, jss)

The exception is all of the following:

• System.NotSupportedException: 'TypeConverter cannot convert from (null).'
• System.NotSupportedException: 'TypeConverter cannot convert from System.String.'
• System.NotSupportedException: 'CollectionConverter cannot convert from (null).'
• System.NotSupportedException: 'CollectionConverter cannot convert from System.String.'

The error isn't technically being thrown by JSON.NET itself, but by .NET and so consequently JSON.NET isn't providing me with any useful information on what the underlying cause is.

I'd post an example class that's being exceptioned on, but they're a bit too involved. But don't forget: they were working completely fine previously.

I didn't make any changes to any of the classes being serialised, so why has this started happening? Who the hell knows. What I do know is that all development has halted.

This is with Newtonsoft JSON.NET 10.0.3.

Edit: It appears I've found one of the causes. It's when using the property attribute DefaultValue. In this case, it's when setting the default value for SystemColors. Here's the code below, in this one particular case.

    Private _customColor As Color = SystemColors.Info

    <Category(""), Description(""), Browsable(True), DefaultValue(GetType(SystemColors), "Info")>
    Public Property CustomColor() As Color
        Get
            Return Me._customColor
        End Get
        Set(ByVal value As Color)
            Me._customColor = value
        End Set
    End Property

Edit 2: Managed to fix all of the FileSieve issues; this means I have a ton of other projects I need to similarly fix. I hope it's just a bug introduced into JSON.NET that stops DefaultValue() from being used. Each complex property now needs a corresponding method that determines if it should be serialised or not.

To fix the above color property, remove the DefaultValue attribute from the property and use the following method within the same class as the property itself:

    ''' <summary>Determines if CustomColor should be serialised depending on if it's the same colour as its default value.</summary>
    ''' <returns><c>True</c> if CustomColor is different to the default and should be serialised, otherwise <c>False</c>.</returns>
    <EditorBrowsable(EditorBrowsableState.Never), Browsable(False)>
    Public Function ShouldSerializeCustomColor() As Boolean
        Return (Not Me._customColor.ToArgb.Equals(SystemColors.Info.ToArgb))
    End Function

The reason for the ToArgb() is as follows:

To compare colors based solely on their ARGB values, you should use the ToArgb method. This is because the Equals and Equality members determine equivalency using more than just the ARGB value of the colors. For example, Black and FromArgb(0,0,0) are not considered equal, since Black is a named color and FromArgb(0,0,0) is not.

All → Software → FileSieve

Just a quick one today as I'm feeling like sheeeeeet.

FileSieve 4.20 is shaping up to be a pretty hefty release as it contains the most changes for the 4 series to date. Because of that, I'm adding more changes, and even more changes. Gonna break a record? Do it properly.

Today's (partial, as I was away for a lot of the day) task is to try and get the Prescanning split into multiple threads.

Before FileSieve does its sieving thing, it has a Prescan stage where it gathers information on all of the files to be processed. This is currently a sequential process and can take a while as it needs to traverse the file system and grab various pieces of metadata.

Now, what if the files reside on different drives? We can portion out all Source Items (a directory, essentially) onto different threads depending on which drive they reside on. This needs a bit more than simply creating a list of Source Items per drive letter due to logical and physical drives. That is, a single drive may have multiple logical partitions, and we don't want to thrash the disk because we're too dumb to realise all of those partitions share the same drive heads.

This Prescan threading feature has been sat dormant in FileSieve for quite a while as I didn't get around to implementing the are-drive-letters-on-the-same-drive logic. Over the next couple of days I'm going to see if I can determine how feasible this is; I definitely do not want to rely on Windows Management Instrumentation as the WMI infrastructure is one bloaty mutha - it takes literal seconds to instantiate for the first time.

All → Software → FileSieve

That's a new version of FileSieve done and released. I think this particular version has the biggest change log so far. It would've been bigger - he said - but I, for some reason, decided to revisit the Script plugin.

I don't think I've gone into this before other than with a small forum post, but I started a new (Method) plugin a while ago that is what amounts to a small programming environment.

The purpose of the Script plugin is to allow the user to program their own Method and Modifier plugins. Full intellisense, code completion, debugging, and all that guff. It's undoubtedly far too much effort for the outcome of only a few people even being able to effectively use it as it requires programming knowledge. And then the documentation. Holy crap - no, I've already been through that nightmare with the original VoiceMate Professional.

While working on the prescanning threads feature, for no explicable reason I decided to jump into the Script plugin and work on that. The plugin all now builds and runs, but then common sense won out in the end and I unloaded the project from the solution and ended it there.

As for the prescan threading: it didn't make it into 4.20. The scanning was never designed from the beginning with concurrency and so random things aren't thread-safe, resulting in Weird Things Happening. I'm not going to delay the new big release because of this, so the threading features were disabled and the new release has been made.

All → Software → FileSieve

As an excuse to use this new keyboard, I was going to rewrite a part of the FileSieve documentation but then I realised I didn't do a blog post for today. So here I am.

Today was spent working on FileSieve and getting command-line automation happening.

As part of my App Framework library, the Parameters class provided all the functionality I needed to get things moving fast. It was simply a case of reiterating over the Parameters.List collection and performing commands (aliases are also supported) based on what was encountered.

Due to this iterative nature, FileSieve can executes multiple commands from the command-line in sequence. So, for example: you can select a profile, start it, and then select another profile and start that, even making changes to it - and only it - beforehand.

The first added command was Profile, which is used like so:

-profile="My Profile"

This makes the GUI select the My Profile profile and make it active. All following commands are then executed on that profile.

Most of the commands emulate using the application's interface, such as RadioButton.PerformClick(). There's additional sanity-checking in there as it's accepting input from the user, but there really isn't much to it.

I did wonder about adding generic automation to the App Framework for all of my software, but it'll still take manual work to specify what's what. Probably isn't worth it, to be honest. Although I think Wake On LAN Ex will benefit from automation.

The 4.22 release will feature additional commands that aren't critical to using FileSieve from the command-line.

Oh, speaking of the command-line: the nomenclature has been changed from Command-Line to Automation. The reason being that I'd like to also add support for automating FileSieve usage via a text file. It'll basically be a top-to-bottom list of commands and parameters that FS iterates through executing one at a time.

To help with any debugging issues on the user's part, all automotive actions and their outcomes are logged under the Automation category in the Log window.

FileSieve 4.21 is available from FileSieve.com.