Library ported to Go

[jimsmart]

Hi,

Just to let folk know that I have ported this library from its original C++ into Go.

https://github.com/jimsmart/grobotstxt

My conversion includes 100% of the original library's functionality, and 100% of the tests.

Because I approached this conversion function-by-function, some of the resulting code is not necessarily idiomatic Go — but, in places, I have made some cleanups, including renaming a few things.

But otherwise my package is a faithful reproduction of the code in this here repo.

I have licensed my code with Apache 2.0, as per this repo, and I have preserved existing copyright notices and licensing within the appropriate source files.

— Regarding this last matter, as my code is technically a derivative of this repo's code, would someone here please care to check my project with regards to the above-mentioned licensing requirements, to ensure that what I have done is correct? Many thanks.

/Jim

[jimsmart]

Additionally, whilst reviewing the original C++, I have managed to implement my equivalent of RobotsTxtParser::Parse so that it does not allocate a 16k buffer for reading lines into [1], by instead using a windowed substring over the string currently holding robots_body_ (in my code I track start and end indexes, see [2])

In Go, this is easy to do, because we have string slicing, so the memory backing the substrings isn't even copied — I strongly suspect that similar should be possible in C++ with absl::string_view (but my C++ is very rusty).

Is this something you folk would be interested in fixing? Should I open another issue to track this?

[1] https://github.com/google/robotstxt/blob/master/robots.cc#L392
[2] https://github.com/jimsmart/grobotstxt/blob/master/robots_cc.go#L471

[Folyd]

Hi @jimsmart . I also ported a Rust version of this library. See issue #31 😸

[nzrsky]

Btw, I made a performance comparison between these ports and the C++ version (including my own C++ fork) https://github.com/nzrsky/robotstxt/blob/master/COMPARISON.md

[garyillyes]

@jimsmart @Folyd

We'd like to highlight community ports. Could you run these through https://github.com/google/robotstxt-spec-test/ and report back?