Jul
22

Assembler, part 4 2017

What am I doing?

I've once again resumed working on the interface rather than the actual core; I still don't even know if any of this truly works. Sheesh.

Not a whole lot to report from yesterday as I practically spent the day reading while doing the odd bit of UI tinkering. Big thanks to Rob for reminding me about SyntaxBox! Scintilla is pretty horrible overall and SB is done so much better, so I'm glad the editor is now more solid. Code folding was extremely easy to implement, unlike Scintilla where I just couldn't be bothered and worked on another part instead.

Today I started a new control for displaying parsed instructions. Think of it as a ListBox but with no user interaction, and the "current item" is always vertically centred and highlighted; currently adding column support to tabulate the view. Oh, issue: I can't get the background transparent no-matter what. I've done millions of transparent controls in the past, so this is confusing the hell out of me.

Next, I need to have a go at actually executing instructions. I've done very little bit-work in .NET so I'm not sure how that's going to go, especially as I don't even know the syntax for VB.Net; presumably it's going to be bat-shit insane compared to C#.

Nothing instead of just null still annoys me to this very day.

Jul
20

Assembler, Part 3 2017

I've gone ahead and added the BaseMnemonic class, along with a IMnemonic interface that all CPU mnemonics (instructions, op-codes - whatever you want to call them) inherit from.

The past hour or so has been spent improving the interface and adding things where necessary. The main window now features has the standard set of root menu items (File, Edit, View, Debug, Tools, Help) and there's a (currently collapsed) project viewer.

Added a Messages window that will show To-do items, syntax errors, and the like. Also added Status Register to the Registers window. The tooltip for each register's value also displays the value of that register's contents in base 10 (Decimal) and likely base 16 (Hex) in the near future, too.

Next up is making use of the BaseMnemonic class by implementing the Move instruction. Oooh!

Messages and Registers windows


Assembler, Part 2

Part number the two!

Did some more work on the Assembler project (part 1 here) that didn't just involve improving the interface. Ooh, ooh - what?

Parsing. That's what.

Hitting Run (or F5) now results in the entered code being parsed and added to an Instructions collection. The following style is correctly parsed:

<mnemonic><datatype> <source_operand>, <dest_operand>

An example of which is:

move.l #4, d0 ; an optional comment which isn't explicitly captured

mnemonic is the instruction (move, add, etc). datatype is the size which is either .b (byte), .w (word), .l (long-word). source_operand and dest_operand is where stuff happens (values, registers, addresses, and the like).

Each line in the source is read one-by-one. Once a line is read, a regular expression1 match is attempted (hard-coded add and move for now) .

(add|move)(?:\.(l|b|w)?)? *(.*), (\w*)

Note that this expression is far from complete. First change would involve removing the rigid white-space structure.

The reason for the datatype being optional in the regex is because I thought it could be omitted from assembler source. Doing a bit of research seems to show that everyone always includes a datatype (or size as I think is the nomenclature), so I've dropped the idea of having a default type. Still, this now means I can detect and specifically give an error about any missing types.

Anyway, back to parsing.

If the line fails the regex match, then onto the next line. If it succeeds, the parsing continues and an Instruction object is created containing mnemonic and datatype. I haven't yet figured out how the source and destination operands are going to be represented. In assembler code, they can be addresses, registers, or whatever else so I don't really know what's going to happen until I just start typing.

Undecided if mnemonics should inherit from a baseMnemonic and be individual classes, or if I should continue with the mnemonic simply being an enum on Instruction. I like instructions being self-contained, but comparisons become more involved. I'm leaning more towards baseMnemonic with a defined Interface so instructions can do whatever the hell they want as long as they take in an input and give back an output.

Thinking further ahead, I'm also undecided if the entire assembler source should be parsed in one go or instead proceed one line at at time. The latter will allow code to be edited while the program is running, which is quite nice, and goes hand-in-hand with the Step debugger feature.

Right now, I can't think of any advantages of parsing in one go (parse each line and populate Instructions immediately before execution even starts). Maybe "compiler" optimisations at a later date? Performance isn't even a minor consideration, so I don't see that as much of a win. I think the step-by-step method simply gives more advantages and so that may very well be the route to take.

1 I can highly recommend Regex Pal for writing and testing expressions.

Jul
19

Assembler, Part 1 2017

I don't yet have a name for the Assembler language program-thingy I'm doing so I'm just going to refer to it as Assembler (giev suggestions!) for the time being. Cool? Cool.

As mentioned in a previous post, I'm taking time out from my commercial software and am instead working on fun little projects. Clearly, our definition of the word fun differs greatly.

I wrote a Turing program a long time ago and recently it occurred to me that I could get a more advanced version by essentially emulating a CPU. The Turing machine is pretty much what that is. In this case, I'd be going with a Motorola 68000. Because it's lovely, and because it seems simple enough as compared to the nightmares that are the x86 and x68 architectures. Bear in mind that I don't really know much about assembler/assembler other than tinkerings back in the late 1990s.

Most of the time so far has been spent creating a lexer (syntax highlighter) for Scintilla. Being a .NET OOP guy and working with Scintilla.NET (which doesn't attempt to really refactor anything), its API is a complete mess. So it took me quite a while to get highlighting happening, and it's still not done. Thankfully there's a project called ScintillaNET-Kitchen that helps out.

I haven't gone for a multi-document interface or anything that you'd find in more complete IDEs - how do I even know if this project will work satisfactorily? Hell, saying that, I've (partly!) made it possible to switch in new CPUs. I think that itself is a bit much and so don't currently plan on fully adding support for such a feature. Hmm... I could also create a chipset with multiple chips...? No - NO. Stop right there.

Yeah, alright - what the eff is the point?

Fun. As said above.

I'm going to keep adding to it. Forever. CPUs don't tend to do a whole lot on their own; their instruction sets aren't very big, either. Lots of shunting of data all over the place. Of course this is simplified and from someone that doesn't really assembler-me-do, but how hard can it be?

So, if I can get the basics working then I can start adding built-in routines that a program can call to do more advanced stuff. A video display would need to be added, of some sort. Only being able to manipulate registers isn't the most exciting thing after the initial novelty wears off.

Shut up and tell me if it's done yet

Hell no it isn't done. It doesn't even parse the entered code yet. There's the editor with all its syntax highlighting tomfoolery, and there's the Registers window that displays the contents of each register. That bit works; it shows exactly what's in the registers, but they can't be set using code yet.

There's a screenshot below. That's all there currently is. Will still if I have time to get the parsing - at least started - in today.

Assembler, screenshot uno

Edit: Did the opposite of what I said and have continued working on the interface. Oops.

Jul
16

FileSieve 4.20, Scripting, and Threads 2017

That's a new version of FileSieve done and released. I think this particular version has the biggest change log so far. It would've been bigger - he said - but I, for some reason, decided to revisit the Script plugin.

I don't think I've gone into this before other than with a small forum post, but I started a new (Method) plugin a while ago that is what amounts to a small programming environment.

The purpose of the Script plugin is to allow the user to program their own Method and Modifier plugins. Full intellisense, code completion, debugging, and all that guff. It's undoubtedly far too much effort for the outcome of only a few people even being able to effectively use it as it requires programming knowledge. And then the documentation. Holy crap - no, I've already been through that nightmare with the original VoiceMate Professional.

While working on the prescanning threads feature, for no explicable reason I decided to jump into the Script plugin and work on that. The plugin all now builds and runs, but then common sense won out in the end and I unloaded the project from the solution and ended it there.

As for the prescan threading: it didn't make it into 4.20. The scanning was never designed from the beginning with concurrency and so random things aren't thread-safe, resulting in Weird Things Happening. I'm not going to delay the new big release because of this, so the threading features were disabled and the new release has been made.

Jul
15

Pre-scanning Threads 2017

Just a quick one today as I'm feeling like sheeeeeet.

FileSieve 4.20 is shaping up to be a pretty hefty release as it contains the most changes for the 4 series to date. Because of that, I'm adding more changes, and even more changes. Gonna break a record? Do it properly.

Today's (partial, as I was away for a lot of the day) task is to try and get the Prescanning split into multiple threads.

Before FileSieve does its sieving thing, it has a Prescan stage where it gathers information on all of the files to be processed. This is currently a sequential process and can take a while as it needs to traverse the file system and grab various pieces of metadata.

Now, what if the files reside on different drives? We can portion out all Source Items (a directory, essentially) onto different threads depending on which drive they reside on. This needs a bit more than simply creating a list of Source Items per drive letter due to logical and physical drives. That is, a single drive may have multiple logical partitions, and we don't want to thrash the disk because we're too dumb to realise all of those partitions share the same drive heads.

This Prescan threading feature has been sat dormant in FileSieve for quite a while as I didn't get around to implementing the are-drive-letters-on-the-same-drive logic. Over the next couple of days I'm going to see if I can determine how feasible this is; I definitely do not want to rely on Windows Management Instrumentation as the WMI infrastructure is one bloaty mutha - it takes literal seconds to instantiate for the first time.

Jul
13

Wake On LAN Ex 3.10 2017

Another day, another release.

There was only really supposed to be one change to this new release:

Performance improvements to the loading and saving of settings.

This is the result of the FileSieve and Newtonsoft JSON.Net findings. I reworked my (JSON/XML) serialisation code and took out the options that allowed you to specify Image and SecureString type conversion, and now instead require that all properties of those types have a TypeConverter attribute specified.

<JsonConverter(GetType(SecureStringConverter))>
Public Property Password As SecureString
    ...
End Property

As well as simplifying the code - we're only talking about a reduction of 4 lines here - it removes the enormous performance drain of JSON.Net's TypeConverter CanConvert property. It does mean you can't just pass the relevant converter option, but with the huge gains - who cares, really?

Sorry, I lost the main point of this point: there are now quite a few changes in this new WoLEX release; once I started, I couldn't stop.

Jul
12

FileSieve and Newtonsoft JSON.NET 2017

After the last Wake On LAN Ex release, FileSieve needs to be sorted out and released itself. This particular 3.20 release contains significant bug fixes that really needs to be put in the hands of FS users.

But when - just as before - I was just about to get the remaining issues fixed, I ran into another issue. A gigantic, showstopping issue that I'm absolutely confounded by.

Any time FileSieve does any (de)serialisation, JSON.NET throws exceptions; specifically, the .NET runtime throws exceptions. FileSieve worked absolutely fine forever - until this started happening yesterday. All of my other software that also (de)serialise are doing the same thing.

The code in-question (VB.NET):

Return JsonConvert.DeserializeObject(Of T)(tr.ReadToEnd, jss)

The exception is all of the following:

  • System.NotSupportedException: 'TypeConverter cannot convert from (null).'
  • System.NotSupportedException: 'TypeConverter cannot convert from System.String.'
  • System.NotSupportedException: 'CollectionConverter cannot convert from (null).'
  • System.NotSupportedException: 'CollectionConverter cannot convert from System.String.'

The error isn't technically being thrown by JSON.NET itself, but by .NET and so consequently JSON.NET isn't providing me with any useful information on what the underlying cause is.

I'd post an example class that's being exceptioned on, but they're a bit too involved. But don't forget: they were working completely fine previously.

I didn't make any changes to any of the classes being serialised, so why has this started happening? Who the hell knows. What I do know is that all development has halted.

This is with Newtonsoft JSON.NET 10.0.3.

Edit: It appears I've found one of the causes. It's when using the property attribute DefaultValue. In this case, it's when setting the default value for SystemColors. Here's the code below, in this one particular case.

    Private _customColor As Color = SystemColors.Info

    <Category(""), Description(""), Browsable(True), DefaultValue(GetType(SystemColors), "Info")>
    Public Property CustomColor() As Color
        Get
            Return Me._customColor
        End Get
        Set(ByVal value As Color)
            Me._customColor = value
        End Set
    End Property

Edit 2: Managed to fix all of the FileSieve issues; this means I have a ton of other projects I need to similarly fix. I hope it's just a bug introduced into JSON.NET that stops DefaultValue() from being used. Each complex property now needs a corresponding method that determines if it should be serialised or not.

To fix the above color property, remove the DefaultValue attribute from the property and use the following method within the same class at the property itself:

    ''' <summary>Determines if CustomColor should be serialised depending on if it's the same colour as its default value.</summary>
    ''' <returns><c>True</c> if CustomColor is different to the default and should be serialised, otherwise <c>False</c>.</returns>
    <Browsable(False), EditorBrowsable(EditorBrowsableState.Never)>
    Public Function ShouldSerializeCustomColor() As Boolean
        Return (Not Me._CustomColor.ToArgb.Equals(SystemColors.Info.ToArgb))
    End Function

The reason for the ToArgb() is as follows:

To compare colors based solely on their ARGB values, you should use the ToArgb method. This is because the Equals and Equality members determine equivalency using more than just the ARGB value of the colors. For example, Black and FromArgb(0,0,0) are not considered equal, since Black is a named color and FromArgb(0,0,0) is not

Jul
10

Imagus Extension for Chrome 2017

Just a quick post to mention an extension for Chrome called Imagus.

When you hover over a link or an image, it'll show you the enlarged version (as if you followed the link), or it'll follow and parse the destination image and show that. It saves you having to manually follow the link and hitting back.

It's bloody useful - give it a try.

Note: The Imagus extension has nothing to do with me; it's just damn useful.


Wake On LAN Ex 3.09

Just like the dopey FileSieve problem, I may have made a total dumb-dumb with Wake On LAN Ex.

That problem where a user reported that WoLEX couldn't wake up a Buffalo TerraStation? It's because they use port 2304 for waking-up, and WoLEX wasn't actually taking any notice if the user entered a port that wasn't the default port 9.

Oh dear.

Thanks to A. Jones for investigating the issue.

New version released.

Jul
09

WoLEX and TeraStations? 2017

Uh-oh, I received a report earlier today saying that Wake On LAN Ex isn't waking up Buffalo TeraStations.

Considering that WoLEX is waking up other devices fine, I'm inclined to believe there could be a configuration problem with said TeraStations, or even the network to some degree.

I've asked if the app is able to wake-up other devices on the network and will decide what to do depending on the answer. If not - or n/a - then I'm not sure what the plan of action is, in all honesty. I could try adding another way of implementing the WoL functionality, but I'm still confused why the existing method would only wake some devices?

Jul
07

FileSieve Settings - Fixed! 2017

Days and days to even work out why plugin settings weren't being set correctly and getting no-where, then this morning, within a couple of minutes of loading the project... it's fixed!

The issue was when a Profile was set, the Profile's active Method ("a method of sorting files") wasn't being initialised (which, among other things, passes the relevant settings on to the plugin). This doesn't make much sense as that's something that must have been happening previously otherwise it wouldn't work in the past. Was it broken all this time? Nineteen releases and no-one's mentioned it? Even I didn't realise?

That can't be right.

But, it's fixed, and I've created a new build and sent it to the user mentioned in previous posts to see if his original issues are fixed. If they are, then it'll be time to release 4.20 - a release all FileSieve users should update to.

Wondering if I should take a little break until I get the go ahead or if I should move this development blog over to the main domain...

Jul
03

FileSieve Settings Insanity 2017

Oh maaaaan.

There's a gigantic issue with FileSieve and its plugins' settings. The wrong settings are being activated for the wrong Methods/Modifiers.

And it's an absolute nightmare to work out why, or where, or anything.

Another go tomorrow, then.

Jul
02

Is Locked? 2017

Rediscovered one of my software projects from the space-year 2011: Is Locked

It's a little program that allows you to drag and drop files into it and it'll tell you if those files are currently locked (either by another program or Windows itself). I made this for myself as I wanted to quickly know which of the multitude of files I was transferring via FTP were complete.

Any unlocked files can be copied or moved to the clipboard (via a true FileDrop drag operation; not just a textual representation of their filenames) ready to be pasted into an Explorer or whatever window.

Needs a little bit more polishing and then I'll kick it out there; I managed to resist redoing the GUI to be a more modern flat-style, otherwise it may not get released again.

There's so many projects that are on the verge of completion but I just haven't bothered sorting out that last couple of percent.


Windows Re-installed!

Alright, Windows itself installed extremely quickly but installing stuff like Visual Studio took forever. It's now all finally done.

There are various issues in FileSieve that need to be fixed before I can release 4.20, so I'll continue work on it tomorrow (a Monday) rather than jumping into it mid-day on a Sunday. The Method/Modifier settings not being loaded at the right time is extremely confusing so will need a huge amount of thought and debugging.

Jul
01

Reinstalling Windows 2017

Alright, this messed-up Windows install is annoying me too much to continue. Time to re-install Windows 10 again to see if it'll actually work properly this time...

Edit: I couldn't find this info anywhere, although it tells you after you click the Delete everything option.

Does the Reset PC -> Remove everything option also wipe/format any other installed drives and partitions?

The answer is: it's up to you.

Windows will tell you that there's more than one drive installed and will ask you if you want to just format the drive Windows is installed on or format all drives.

Now you know.

Edit 2: Damn Windows installs fast. 6-8 minutes, maybe?

Jun
30

Broken Chrome and Windows Re-installation 2017

As rare as Windows (10) re-installations are these days, a fresh installation being immediately broken has happened at least twice now.

Some applications can't save their settings and Google Chrome now won't open most Google services due to NET::ERR_CERT_AUTHORITY_INVALID and HSTS errors.

I'm supposed to be doing some major FileSieve work over the next few days!

Edit: More information on the Chrome issue.

But first, system information: Windows 10 x64 (Creator's Update), GeForce GTX 770, Skylake i7 6700K CPU, 32GB RAM; all updated drivers

The cert problem in Chrome is something I've seen a bit too often over the years; with no change in the OS environment, Chrome will start giving the NET::ERR_CERT_AUTHORITY_INVALID error, along with a message that the site can't be seen at all due to it using HSTS. But this is new in that its specifically the Google services, and Gmail's missing icons.

Microsoft Edge works without issues. For the first time - ever.

This is happening for YouTube, the Chrome Store; not sure if it's affecting non-Google sites as I haven't browsed many today. One difference, though, is Gmail works but its web-fonts(?) are blank. Uh, how do I show a screenshot in Markdown...

Gmail: no icons

The vertical lines denote where I snipped the image to reduce its width.

The screenshot shows that various within-button icons are entirely missing, including the images displayed within each tab.

Additionally, the Hangouts Chrome extension won't sign-in.

Edit 2: Argh, re-installed Windows and everything works fine - except Chrome. It's now complaining with the same error above (seems it's when I restart Windows?), but thankfully I'm able to fix it via the following:

Fully exit Chrome via the systray icon
Open a command prompt in Windows
Type `ipconfig /flushdns`, and hit Enter
Open Chrome and it works correctly

Jun
28

Cores, Plugins, and Rewrites 2017

What a frantic couple of hours.

A user - the same as the previous post(s) - reported another couple of issues involving FileSieve, audio tag data, and Windows path and file names.

The first being that any audio files with a genre with a slash character (which is invalid in a Windows path - when not used a path separator - or filename) wasn't having those slashes replaced with a valid delimiter. This was an easy fix as the slashes were correctly being replaced, but any files that had multiple genres weren't having a delimiter appended after each genre (with the last dangling delimiter being stripped off the end), so you'd end up with a genre of FolkPop as opposed to Folk, Pop.

The second issue involves the way Windows treats directory - and probably file - names that end in a full-stop or a full-stop and a space. To put it bluntly: it doesn't bloody well like it.

FileSieve creates a directory based on a file's audio tag data, such as (minus quotes) D.I.S.C.O. but Windows internally, somewhere, "corrects" that to D.I.S.C.O thus stripping off the trailing full-stop. If there was a space after the full-stop, then it removes that, too.

Of course FileSieve doesn't see this ninja-change that happened internally within Windows and then chucks an error into the log saying that it couldn't find the file. That's because Windows stuck its D in our A.

Then I ran into an issue while beginning to debug the above path issue. Of course I did.

The (audio) Tag and (photo) EXIF plugins. They allow a mask to be set, such as %ALBUM_TITLE%\%TRACK%. %TITLE%.%FILE_EXTENSION%. When an (audio) file is processed, it would be placed into a directory and named as follows:

My Album\01. My Song.mp3

Except it doesn't. The file isn't actually renamed at all; the directory structure is created, but the filename is kept the same as the original.

This, as initial impressions appear, may be due to a core change made within FileSieve's processing. Plugins set a processed item's Path property independently of its Filename property. This means that plugins don't need a pile of additional logic to determine and set an item's path and filename. FileSieve's core will do that.

Unfortunately, it appears the Tag and EXIF plugins weren't updated to make use of this change and so treat both the path and file names as one property. Tomorrow, this changes.

Jun
27

Wonky Path Handling: Fixed! 2017

Finally got to the bottom of the issue that user was having with directory names in FileSieve.

Turns out it was due to a Modifier plugin affecting the entire path of items rather than just the path after the root destination.

FileSieve creates directories within a destination the user specifies, but the Find Replace modifier plugin was performing its modifications on the entire path, including the drive letter. There's absolutely no point to this (it was an oversight, originally - now corrected) as the destination already exists and cannot be modified (eg. changing case).

This user was using the Find Replace modifier in a really good way - replacing invalid characters with valid ones so FileSieve doesn't skip out on them. But using the previous (incorrect) entire-path-is-affected replacement style replaced the : colon character (as part of the drive letter) with a suitable replacement as specified by the user.

Oops. Now fixed; hopefully verified by the user himself as I've done a test build. A full release will be made shortly after verification.

While I was working with FileSieve anyway I also fixed some other stuff and re-instated the Invalid Filename Editor to replace any invalid characters encountered within filenames - a more consistent and low-level version of the Find Replace substitution method above.


Invalid Filename Editor

Invalid characters in filenames suck.

Any plugins in FileSieve that return an invalid character results in the file the plugin is working on being skipped due to a failure when combining paths together. This is a controlled failure, I suppose you could say, but it's a failure nonetheless.

Despite adding an Invalid Filename Editor to the previous version of FileSieve, I disabled it as I wasn't sure at which step invalid characters should be substituted. Well, due to a quirk/bug involving filenames as reported by a user, I had another go at getting this working again.

Substitution now happens before path combining occurs, which I'm not 100% sure is the correct location. More of a semantic thing, but I'd like to move it to another step - not easy to do as this occurs during a (project-wide) plugin that does not reference the main, parent application.

Jun
26

Modifier: Find Replace 2017

A couple of new categories for the site!

Today someone actually registered on the Tracker.BootBlock.co.uk site for the first time, and as a bonus, they even created a nicely detailed bug report.

So I began working on fixing the bug in FileSieve (specifically, a plugin) and instead got side-tracked and added a preset set of filters for removing invalid OS characters from filenames.

Next up will be to add low-level support for removing/replacing invalid characters rather than shoving it into a plugin, and fixing the reported bug.

Edit: The date grouping issue on this site doesn't look like it has been fully resolved.