DIARY.txt

This file is a diary of my progress.

Times are given in GMT, I'm currently in La Paz, Bolivia, which is
GMT-4. So the competition started at a nice eight o'clock in the
evening.

Sitting in the bar of the hostel where I live.

I realise that I have a lot of functions that could use an accumulator
and be tail recursive.  They run in constant time so I can't be
bothered.  I.e. it's only the code generator that has that type of
code, and the XML files are a known constant size.

2011-07-09 00:00

Creating directory structure.

Copy of basic build files from another project.

Adding lots of .gitignore files.

Copying xml files from quickfix code. Running 'for a in FIX*; do mv $a
$(echo $a | tr A-Z a-z); done' to make the files lowercase.

I know that I should probably add a COPYRIGHT file related to these
files, but that will have to be handled later. (QuickFIX has a very
lax license; don't worry.)

00:24

Cheese burger ordered.

Adding a fix_generate.erl stub file.

Adding Rules.mk for priv directory.  Which will eventually create one
fix parser file for each xml file.  I realise while I have done
autogenerated things in Makefiles before they have only been a single
file thus filenames have been hard coded.

Once again learning to use .SECONDEXPANSION the limitations it has.  I
never again want to write a dependency like $(d)/$$(basename
$$(@F)).xml or generate a file named $(@D)/../src/$(basename
$(@F)).erl

Still not sure whether the generating Rules.mk file should be in the
priv directory or the src directory.

Build broken.

01:04

Really realising that I've never used xmerl before.  Code looks ugly
as hell.

01:15

Stupid make dependencies.

Apparently the Rules.mk file should go in the src directory.

01:43

Commiting a code generator that generate files that compile. YAY!

Milestone 1 reached.

Second half of Peru vs. Mexico in Copa America is about to start.

Starting to write a test script.

03:00

Have a decent test script up and running.

People at the bar are very happy.  So am I, if for a completely
different reason. (Local time is 23:00)

Writing more fix_generate stuff.  Not really doing any progress.  This
has to change.

04:43

Headers can now be parsed in theory.  But only if they are
typeless. :-)

05:20

Dancefloor next to computer managed to grab my attention.

06:10

Back to laptop, with a quick solution to earlier problem.  How great
it is to take a break sometimes.

First full message has been parsed.  Yay!  Happy time!  I would
celebrate with a beer, except I just drank a Vodka-Coke.

07:08

Heading for bed.  (Local time is 03:08)

14:51

Computer up and running.  Eating breakfast.

Downloading specifications from the FIX webpage.  The definition of
data types change over versions.

16:43

The more I read my code the more I think that xmerl must have
functionality for what I do with list comprehension.  I wonder if that
would look as ugly, worse, or better.

Working on handling body parsing.  And enum handling.

The more I write the more I understand why I did not want to write
this in XSLT.  It's bad enough in Erlang.  :-)

17:32

Yay!  Code builds again.

Next step is to do some cleanup before continuing.

18:58

Basice cleanup done.  Code starts to be readable.

Working furnther on creating a parser for the trailer. Three lines of
code.  I love code reuse.  That's why the cleanup was good.

20:39

Unfortunately I've done some assumptions on typing which was a bit
bad.  Fixed but doesn't work for FIXT.  The reason is that FIXT has
the header/trailer defined in the fixt xml file while the message body
is in fixt50*.xml.

Crap.

Other problem is that I'm currently only matching enums which are
strings.  Need to look at the type when I generate the code.

20:42

Food.

24:00

Half time.

I've done something stupid with my makefile.  It builds too much.  I
guess it's better than the other way around.  Mark that as won't fix.

Working on building a bunch of Erlang records so that I can return
records instead of ugly Key, Value pairs.

YAY!  Found why my makefile was playing havoc on me when adding
generation of hrl files.  My autogenerated files dependend on .d files
(which are autogenerated things related to Erlang dependencies), and
to be able to create the .d file we first need to create the .erl file
of course.  Stupid.  Fixed.

01:11

627 lines of Erlang code.  From 25 hours of coding.  Hmmm.

A smidgen above 2000 lines of generated code though.

I have no idea whether this is good or bad.

01:32

Food break.

02:09

Back with a full stomach.  Great!

Hopefully this means less crappy code... :-)

04:38

Yay!  I'm back where I was a few hours ago.  Except now I return a
record instead of key value pairs.

04:45

My line count has gone down.  I counted a file I shouldn't have.

So 519 lines of written code.  76804 lines of generated code.  I
wonder if I can decrease those 519 lines further.

There are probably speedups possible.  The code generation is really,
really slow.  On the other hand, it's XML, and I don't really do any
optimisations with regards to accessing items in the structure.

07:34

Starting to be tired (local time 03:34).  And I don't really have that
much "easy code" left to write.  Maybe go on tomorrow instead.

My laptop is on it's knees when it comes to generating and compiling.
68000 lines of generated code.  And I don't really see a good way to
make it shorter.  There are minor differences between versions so
shared files are hard to make.

07:44

82461 lines with checking for mandatory fields.  Ouch!

08:00

Time for bed.

14:11

Time for breakfast.  Less than 10 hours to go.

It seems like all almost all the tests "pass".  I use scare quotes
since most of the tests are made to fail.  The ones that should fail,
fail with a throw, which is something which is easily converted to
a decent error message to the client.

15:14

Doing some cleanup and writing basic documentation for the #spawnfest
judges.

16:48

Have fixed a bunch of bugs.  And made the tests better.  A lot more of
them succed when I add a checksum manually.  Should probably add
length manually as well.

I'm doing anything I can to not implement component handling, so I
guess it's time to do that now.

17:31

Stupid repeating groups.  And components.

18:42

Everything seems to work fine.

Only major thing still to do is repeating groups.  I wonder if I can
get faster compile times cause it's starting to be silly.

Files written by me:

> cat fix.erl fix_* | wc -l
596

Files generated:

> cat fix[0-9]* fixt* ../include/fix_* | wc -l
90222

19:17

With repeating groups handled (albeit in a crappy way) the record
definition file is up to 24k lines.  Ouch.

20:40

Lost my motivation.  It works reasonably well.  Feels like the next
step is to pair this thing with a trading platform and find out all
the things that don't work.

21:00

Thinking of ways to make the generated code smaller.  It's silly to
have 24k lines of record definitions.

Maybe the original key, value version wasn't som bad after all.

Cheese burger!

21:20

Laptop randomly decided to hibernate.  Probably because of power cable
glitch. :-(

22:00

Two hours to go.

22:30

Starting to see how repeating groups can be implemented.  And a way to
handle components better.  Probably not possible during Spawnfest
though.  Will decrease that crazy 24k line record definition file.

I could start with making separate records/parsing for components and
see how much that helps.  If it doesn't then the next step is to go
back to key-value lists for non required fields, and only have record
fields for required fields.

23:30

Working hard to make repeating groups work properly (intead of
brokenly as it is now).  Not going to make it I think.

23:38

Found a bug!

Testing takes a very long time, so we'll see if I dare commit the fix.

23:54

I'll go the chicken way.

23:55

No more coding or testing or anything.

Good bye.  See you next year.

Cheers,

Daniel