Yeastplume - Progress update thread - Mar 19 to Sep 19


Update Friday March 1st, 2019

Now working under the results of the last funding round, which I’m celebrating by starting a new thread.

I’ll start off by posting a pic of the latest Windows build running:

Looks extremely similar to the picture I posted a few weeks ago. However, there are many many differences between this picture and that picture which represent most of the past couple of weeks worth of work.

Most importantly, preview windows binaries are now available here:

The server itself (built from a branch on my own repo):

And the wallet:

Grin Windows (Gwin?) now has most of the major issues worked out, and the resulting binaries are now being produced via travis CI. Getting the builds themselves working was an extraordinarily fiddly process, as were just about all of the other issues that needed to be addressed. There’s a pretty decent log of them in the meta-issue, so I’ll leave thumbing through them as an exercise for the dedicated reader.

The largest single change was the dynamic allocation of LMDB backend storage. DB management needed a bit of attention anyhow, and the improvements are completely cross-platform. The old code was just allocating a max db size of 500GB, which was okay on unixy systems where it’s just a max size and not the actual size on disk. Windows being windows needs to allocate the whole thing at once. So that’s been changed to resize the max db size in 128mb chunks as needed on all platforms.

I also more or less finished splitting off grin-wallet, got all builds working on linux/macOS and windows, and removed all trace of wallet code from grin’s 1.1.0 branch. Again, a lot of small fiddly things (such as tests having to be moved/rewritten in some cases,) but nothing too remarkable.

So nothing earth shattering here. Lots of painful work resulting in some big wins for Grin. The wallet is split, windows is working and we can provide binaries. By all means give the windows binaries a try, but keep in mind they’re still pre-release and use at your own risk until we say otherwise. This should all become official for the upcoming 1.1.0 release, at which that point official 1.1.0 binaries will be released (also use at your own risk).

Very glad to have these two things mostly boxed off, and hopefully I should be able to get back to some actual enhancements from next week onwards. Enjoy the weekend.


Update Friday March 9th, 2019

With the windows work and wallet split mostly out of the way and waiting (expectantly for the release of 1.1.0), my focus is back onto improving and extending the wallet core functionality.

In case you’re new to Grin, our strategy with regards to the wallet is not to create the slickest, most polished and user friendly Grin wallet in existence (though we’ll certainly be improving it over time,). It is rather to create the slickest, most polished and developer-friendly Grin wallet toolkit in existence, that will allow community wallet developers to focus on creating the slickest, most polished and user-friendly Grin wallets in existence. grin-wallet is intended to be a set of tools and libraries with a reference CLI wallet wrapped around it.

I’ve started to put down all of the tasks and changes that need to happen to start moving the wallet towards ever-increasing developer friendliness in a meta issue here (roadmap, even?). I won’t repeat the contents of that, but if you’re a wallet developer or at all interested in the direction of Grin’s core wallet functionality, I invite you to have a look and comment.

Most of this past week has been about reviewing all of the various changes, proposals and PRs out there and starting to think about them (I’ll be weighing in shortly in all of the appropriate threads and PRS). I’m also encouraging everyone to get everything prepared for the V2 wallet api tasks, as that’s the next big chunk of work I plan to get involved in.

Other major bits and pieces this week:

  • Fixes to LMDB paths on 1.1.0, turns out they’re a bit all over the place but 1.1.0 should remain consistent with 1.0.2
  • Changes to wallet info display and wallet check should allow wallet users to better differentiate between Unconfirmed coins and coins waiting for the other participant to post the transaction. It should also make wallet check less destructive by default, meaning it won’t wipe pending transactions. Details in issue

That’s it for now, and I look forward to jumping into V2API work next week!

1 Like

Update Friday, March 15th 2019

It’s St. Patricks weekend coming up, so don’t expect much coherence from me over the long weekend. Here’s what went on last week in the continuing saga of grin-wallet.

A couple of major pieces of work are underway. First, I’ve done quite a bit of work on slate version conversions to allow newer clients to communicate with older clients. This works by maintaining each slate version separately, and maintaining code that upgrades and downgrades versions, so if a slate v0 comes in and the wallet is on slate V2, it converts V0->V1->v2, operates on it, then (provided the incoming version is compatible,) converts back V2->V1->V0 to send it back. It’s quite a bit of code to maintain for each version and slightly inefficient, but at least it means that every new slate version just needs a single set of conversions (n-1 -> n, n -> n-1) to ensure backwards compatibility. That’s in place (but needs testing,) which means I can modify the v2 slate with impunity.

Second, I’ve turned my attentions to the V2 API. I have to give a shout out to Andrew Dirksen here, who’s put together an elegant and as-concise-as-possible system for generating the API, which also generates documentation that also acts as integration tests. To support this, he also put together a simplified json-rpc lib that I think is badly needed in the rust world. Now that I’ve had a chance to go over his work in detail, I’m very excited about this and think we’re going to have a very robust, consistent, flexible and self-documenting API v2 that should support all sorts of clients and suit everyone’s needs.

I’ve done some work towards completing the API, but in doing this I started to realize that in order to best take advantage of all of the good things Andrew’s approach to generating API v2 will bring, quite a bit of the code needs to be refactored. I’ll go over the exact changes I’m currently working on either in the issue or in a readme that explains the crate layout, but for now I’ll just say the current code that instantiates concrete wallet support structs and calls the api can definitely benefit from a better separation of concerns. Once this work is done, the code should be much cleaner, but also (very importantly) allow v2 api documentation tests that actually exercise the wallet code and act as complete integration tests as well.

So that’s where things are… naturally the code will need quite a bit of testing once this is done before 1.1.0 can be released, but the work is definitely worth it.

That’s it for now. Remember to enjoy Guinness responsibly.


Update Friday March 22, 2019

Tons of effort over the past week to get these working:

Those are images of the rustdoc documentation I’ve been putting together for the v2 wallet API, and the self-testing documentation for the generated JSON-RPC stubs.

It looks like simple rustdoc documentation, but there’s a lot going on behind the scenes to produce this. That ‘json_rpc_example’ in the image isn’t just documentation, it’s the input to an entire integration test that runs against a sample chain in the wallet’s test framework, with the output carefully compared to the expected results in the documentation. By ensuring documentation and testing are integrated this way, we can pretty much guarantee that both the rust and generated JSON RPC APIs will stay consistent, current and documented going forward. Hugely important, and definitely worth the work.

And what work it is. It goes something like this:

  • Sit down to create the json doc/test for a particular function
  • Realize function args aren’t that json-friendly, fix them
  • Come across obscure bug in serialization preventing example from working, spend hour tracking down and fixing
  • Realise fix has to go into grin core, create that PR
  • Realise function needs a statically-seeded test RNG to produce consistent output every time, implement that,
  • Realise doctest helper functions need more work to support this particular function, implement and test that
  • It’s 4 hours later and I’m still working on documenting the same function.
  • etc, etc, etc.

It’s been like this for just about every single function, and it’s only just been since I got to the last few functions of the owner API that the testing started to go a bit smoother, as most of the bugs and changes required for testing have been ironed out. The upside is that this has been a great test of the V2 API generation, and this should be invaluable going forward to keep the V2 API clean, consistent and documented. The downside is it’s tedious as hell and I can’t stand looking at it anymore right now. Luckily, it’s mostly there, with most API functions and their doctesting complete. I’ll come back and complete when I’m more fresh, but luckily this big push to get the API documentation started should only have to happen once. Doc and test changes from here on out should be incremental and more focused, and hence easier to bear.

Right, I need to go do something that’s not aligning JSON brackets. Enjoy weekend!


Update Friday, March 29th, 2019

Had a good time at Grin Amsterdam during the early part of the week, all a bit of a blur but there’s plenty of video and other documentary evidence from the event. I can never watch videos that include myself, so I’ll leave the simple job of tracking them down to any interested readers.

As for the rest, I’m just in the process of merging a final v2 PR finishing up the API documentation and making a few tweaks and changes to make it as friendly as possible. What’s there after this PR is merged should more or less be what the V2 API will look like for 1.1.0. It is by no means perfect, and minor version numbers (1.2.x, 1.3.x etc) with additions and changes will hopefully be coming on a more frequent basis than the main Grin executable. However, all of this work the past few weeks was vital to ensure changes will be (more or less) self-testing and as self-documenting as possible, which should greatly benefit all future work.

So short terms tasks are now:

  • Wire the rpc-json API into the http listener (so there should be a v2 api listener on the endpoint as well as the previous)
  • Merging grin’s master branch into the 1.1.0 branch, to prepare for 1.1.0 testing
  • Much testing of the wallet 1.1.0 branch to ensure things aren’t completely broken (though hopefully the amount of code not covered by unit testing is significantly shrunk)
  • … All other bits and pieces related to the release of 1.1.0

Looking very forward to getting all that done… see you next week!


Update Friday April 5th, 2019

Going to be a very short and quick update as Mrs. Yeastplume has gone away with several yeastlings so there’s debauchery to be had. All the short term points from last week have been addressed, and we have a 1.1.0 beta which I urge everyone to try. The past week has basically been getting that prepared and then testing away, and a fuller description of the release and how you can help with the testing here:

Have a good weekend, I know I will. (So long as Mrs. Yeastplume retains her habit of never reading these things anyhow).


Update Friday, April 12th 2019

A quiet week from me PR wise, mostly due to the (hopefully) imminent release of 1.1.0 and my own need to do some testing and sanity checking to ensure that 1.1.0 doesn’t have any obvious problems. From what I can gather I think it looks okay, and I haven’t heard about any major issues from anyone (which I hope is due to an absence of serious issues rather than an absence of people trying out 1.1.0). I’m keen to get a release done soon so we can get a baseline in place and start branching so there’s a place for new feature development, as well as remove the wallet code from the grin server master so people stop submitting PRs to it :smiley:

So, given I’m light on detail this week, I thought I’d take a few minutes to organize and relay all of the upcoming tasks that swirl around in my head during the night, many of which I hope should be coming fairly quickly after 1.1.0 with the V2 API and (more importantly) supporting framework out in the wild.

  • Invoicing, i.e. recipient-initiated transactions, should be straightforward and will do next after 1.1.0 release.

  • Fixes and improvements based on the ongoing audit effort, which should be focusing on the wallet right about now.

  • Full wallet lifecycle management via API - I’d like to get to the point where a wallet can be created and instantiated fully through the API, but some thorny issues around password management exist around this that need to be thought through and discussed.

  • More interactive command-line client, i.e. one that starts up and accepts commands as opposed to one-shot invocations (as wallet713 does).

  • Client API Generation, would like to be able to generate client code to call the V2 API in several languages (but could be a big project, just javascript would be a good start)

  • Proof of payment, something that’s being actively discussed but I don’t think anyone has come across the perfect solution. I’m very much in favor of keeping a light touch here and just providing enough to enable the community to build in their own solutions, so we’ll see where these discussions take us.

  • And as an added bonus, I think the Grin server API itself should be redone using the wallet V2 API as a model. May be a good project for someone else to get involved in

I’m looking forward to working on some of these in the coming weeks, and by the way, many of these issues are fairly compartmentalized, and could be very good tasks for anyone looking to get involved with development and wanting to start getting up to speed with Grin.

That’s it for now, enjoy weekend and see you at the next dev meeting!