Yeastplume - Progress update thread - Mar 19 to Sep 19

#1

Update Friday March 1st, 2019

Now working under the results of the last funding round, which I’m celebrating by starting a new thread.

I’ll start off by posting a pic of the latest Windows build running:

Looks extremely similar to the picture I posted a few weeks ago. However, there are many many differences between this picture and that picture which represent most of the past couple of weeks worth of work.

Most importantly, preview windows binaries are now available here:

The server itself (built from a branch on my own repo):

And the wallet:

Grin Windows (Gwin?) now has most of the major issues worked out, and the resulting binaries are now being produced via travis CI. Getting the builds themselves working was an extraordinarily fiddly process, as were just about all of the other issues that needed to be addressed. There’s a pretty decent log of them in the meta-issue, so I’ll leave thumbing through them as an exercise for the dedicated reader.

The largest single change was the dynamic allocation of LMDB backend storage. DB management needed a bit of attention anyhow, and the improvements are completely cross-platform. The old code was just allocating a max db size of 500GB, which was okay on unixy systems where it’s just a max size and not the actual size on disk. Windows being windows needs to allocate the whole thing at once. So that’s been changed to resize the max db size in 128mb chunks as needed on all platforms.

I also more or less finished splitting off grin-wallet, got all builds working on linux/macOS and windows, and removed all trace of wallet code from grin’s 1.1.0 branch. Again, a lot of small fiddly things (such as tests having to be moved/rewritten in some cases,) but nothing too remarkable.

So nothing earth shattering here. Lots of painful work resulting in some big wins for Grin. The wallet is split, windows is working and we can provide binaries. By all means give the windows binaries a try, but keep in mind they’re still pre-release and use at your own risk until we say otherwise. This should all become official for the upcoming 1.1.0 release, at which that point official 1.1.0 binaries will be released (also use at your own risk).

Very glad to have these two things mostly boxed off, and hopefully I should be able to get back to some actual enhancements from next week onwards. Enjoy the weekend.

14 Likes
#6

Update Friday March 9th, 2019

With the windows work and wallet split mostly out of the way and waiting (expectantly for the release of 1.1.0), my focus is back onto improving and extending the wallet core functionality.

In case you’re new to Grin, our strategy with regards to the wallet is not to create the slickest, most polished and user friendly Grin wallet in existence (though we’ll certainly be improving it over time,). It is rather to create the slickest, most polished and developer-friendly Grin wallet toolkit in existence, that will allow community wallet developers to focus on creating the slickest, most polished and user-friendly Grin wallets in existence. grin-wallet is intended to be a set of tools and libraries with a reference CLI wallet wrapped around it.

I’ve started to put down all of the tasks and changes that need to happen to start moving the wallet towards ever-increasing developer friendliness in a meta issue here (roadmap, even?). I won’t repeat the contents of that, but if you’re a wallet developer or at all interested in the direction of Grin’s core wallet functionality, I invite you to have a look and comment.

Most of this past week has been about reviewing all of the various changes, proposals and PRs out there and starting to think about them (I’ll be weighing in shortly in all of the appropriate threads and PRS). I’m also encouraging everyone to get everything prepared for the V2 wallet api tasks, as that’s the next big chunk of work I plan to get involved in.

Other major bits and pieces this week:

  • Fixes to LMDB paths on 1.1.0, turns out they’re a bit all over the place but 1.1.0 should remain consistent with 1.0.2
  • Changes to wallet info display and wallet check should allow wallet users to better differentiate between Unconfirmed coins and coins waiting for the other participant to post the transaction. It should also make wallet check less destructive by default, meaning it won’t wipe pending transactions. Details in issue

That’s it for now, and I look forward to jumping into V2API work next week!

1 Like
#9

Update Friday, March 15th 2019

It’s St. Patricks weekend coming up, so don’t expect much coherence from me over the long weekend. Here’s what went on last week in the continuing saga of grin-wallet.

A couple of major pieces of work are underway. First, I’ve done quite a bit of work on slate version conversions to allow newer clients to communicate with older clients. This works by maintaining each slate version separately, and maintaining code that upgrades and downgrades versions, so if a slate v0 comes in and the wallet is on slate V2, it converts V0->V1->v2, operates on it, then (provided the incoming version is compatible,) converts back V2->V1->V0 to send it back. It’s quite a bit of code to maintain for each version and slightly inefficient, but at least it means that every new slate version just needs a single set of conversions (n-1 -> n, n -> n-1) to ensure backwards compatibility. That’s in place (but needs testing,) which means I can modify the v2 slate with impunity.

Second, I’ve turned my attentions to the V2 API. I have to give a shout out to Andrew Dirksen here, who’s put together an elegant and as-concise-as-possible system for generating the API, which also generates documentation that also acts as integration tests. To support this, he also put together a simplified json-rpc lib that I think is badly needed in the rust world. Now that I’ve had a chance to go over his work in detail, I’m very excited about this and think we’re going to have a very robust, consistent, flexible and self-documenting API v2 that should support all sorts of clients and suit everyone’s needs.

I’ve done some work towards completing the API, but in doing this I started to realize that in order to best take advantage of all of the good things Andrew’s approach to generating API v2 will bring, quite a bit of the code needs to be refactored. I’ll go over the exact changes I’m currently working on either in the issue or in a readme that explains the crate layout, but for now I’ll just say the current code that instantiates concrete wallet support structs and calls the api can definitely benefit from a better separation of concerns. Once this work is done, the code should be much cleaner, but also (very importantly) allow v2 api documentation tests that actually exercise the wallet code and act as complete integration tests as well.

So that’s where things are… naturally the code will need quite a bit of testing once this is done before 1.1.0 can be released, but the work is definitely worth it.

That’s it for now. Remember to enjoy Guinness responsibly.

6 Likes
#12

Update Friday March 22, 2019

Tons of effort over the past week to get these working:

Those are images of the rustdoc documentation I’ve been putting together for the v2 wallet API, and the self-testing documentation for the generated JSON-RPC stubs.

It looks like simple rustdoc documentation, but there’s a lot going on behind the scenes to produce this. That ‘json_rpc_example’ in the image isn’t just documentation, it’s the input to an entire integration test that runs against a sample chain in the wallet’s test framework, with the output carefully compared to the expected results in the documentation. By ensuring documentation and testing are integrated this way, we can pretty much guarantee that both the rust and generated JSON RPC APIs will stay consistent, current and documented going forward. Hugely important, and definitely worth the work.

And what work it is. It goes something like this:

  • Sit down to create the json doc/test for a particular function
  • Realize function args aren’t that json-friendly, fix them
  • Come across obscure bug in serialization preventing example from working, spend hour tracking down and fixing
  • Realise fix has to go into grin core, create that PR
  • Realise function needs a statically-seeded test RNG to produce consistent output every time, implement that,
  • Realise doctest helper functions need more work to support this particular function, implement and test that
  • It’s 4 hours later and I’m still working on documenting the same function.
  • etc, etc, etc.

It’s been like this for just about every single function, and it’s only just been since I got to the last few functions of the owner API that the testing started to go a bit smoother, as most of the bugs and changes required for testing have been ironed out. The upside is that this has been a great test of the V2 API generation, and this should be invaluable going forward to keep the V2 API clean, consistent and documented. The downside is it’s tedious as hell and I can’t stand looking at it anymore right now. Luckily, it’s mostly there, with most API functions and their doctesting complete. I’ll come back and complete when I’m more fresh, but luckily this big push to get the API documentation started should only have to happen once. Doc and test changes from here on out should be incremental and more focused, and hence easier to bear.

Right, I need to go do something that’s not aligning JSON brackets. Enjoy weekend!

7 Likes
#15

Update Friday, March 29th, 2019

Had a good time at Grin Amsterdam during the early part of the week, all a bit of a blur but there’s plenty of video and other documentary evidence from the event. I can never watch videos that include myself, so I’ll leave the simple job of tracking them down to any interested readers.

As for the rest, I’m just in the process of merging a final v2 PR finishing up the API documentation and making a few tweaks and changes to make it as friendly as possible. What’s there after this PR is merged should more or less be what the V2 API will look like for 1.1.0. It is by no means perfect, and minor version numbers (1.2.x, 1.3.x etc) with additions and changes will hopefully be coming on a more frequent basis than the main Grin executable. However, all of this work the past few weeks was vital to ensure changes will be (more or less) self-testing and as self-documenting as possible, which should greatly benefit all future work.

So short terms tasks are now:

  • Wire the rpc-json API into the http listener (so there should be a v2 api listener on the endpoint as well as the previous)
  • Merging grin’s master branch into the 1.1.0 branch, to prepare for 1.1.0 testing
  • Much testing of the wallet 1.1.0 branch to ensure things aren’t completely broken (though hopefully the amount of code not covered by unit testing is significantly shrunk)
  • … All other bits and pieces related to the release of 1.1.0

Looking very forward to getting all that done… see you next week!

4 Likes
#16

Update Friday April 5th, 2019

Going to be a very short and quick update as Mrs. Yeastplume has gone away with several yeastlings so there’s debauchery to be had. All the short term points from last week have been addressed, and we have a 1.1.0 beta which I urge everyone to try. The past week has basically been getting that prepared and then testing away, and a fuller description of the release and how you can help with the testing here:

Have a good weekend, I know I will. (So long as Mrs. Yeastplume retains her habit of never reading these things anyhow).

3 Likes
#17

Update Friday, April 12th 2019

A quiet week from me PR wise, mostly due to the (hopefully) imminent release of 1.1.0 and my own need to do some testing and sanity checking to ensure that 1.1.0 doesn’t have any obvious problems. From what I can gather I think it looks okay, and I haven’t heard about any major issues from anyone (which I hope is due to an absence of serious issues rather than an absence of people trying out 1.1.0). I’m keen to get a release done soon so we can get a baseline in place and start branching so there’s a place for new feature development, as well as remove the wallet code from the grin server master so people stop submitting PRs to it :smiley:

So, given I’m light on detail this week, I thought I’d take a few minutes to organize and relay all of the upcoming tasks that swirl around in my head during the night, many of which I hope should be coming fairly quickly after 1.1.0 with the V2 API and (more importantly) supporting framework out in the wild.

  • Invoicing, i.e. recipient-initiated transactions, should be straightforward and will do next after 1.1.0 release.

  • Fixes and improvements based on the ongoing audit effort, which should be focusing on the wallet right about now.

  • Full wallet lifecycle management via API - I’d like to get to the point where a wallet can be created and instantiated fully through the API, but some thorny issues around password management exist around this that need to be thought through and discussed.

  • More interactive command-line client, i.e. one that starts up and accepts commands as opposed to one-shot invocations (as wallet713 does).

  • Client API Generation, would like to be able to generate client code to call the V2 API in several languages (but could be a big project, just javascript would be a good start)

  • Proof of payment, something that’s being actively discussed but I don’t think anyone has come across the perfect solution. I’m very much in favor of keeping a light touch here and just providing enough to enable the community to build in their own solutions, so we’ll see where these discussions take us.

  • And as an added bonus, I think the Grin server API itself should be redone using the wallet V2 API as a model. May be a good project for someone else to get involved in

I’m looking forward to working on some of these in the coming weeks, and by the way, many of these issues are fairly compartmentalized, and could be very good tasks for anyone looking to get involved with development and wanting to start getting up to speed with Grin.

That’s it for now, enjoy weekend and see you at the next dev meeting!

3 Likes
#18

Update Friday 26th April, 2019

Apologies for missing last week’s update, but it fell through the cracks with all of the school holiday interruptions (how I hate them). Some good development action the past couple of weeks, with a few more things squeezed in for 1.1.0 (which should hopefully drop over the next couple of weeks!):

  • First, a large messy merge update of master branch into 1.1.0, which we then decided to merge back into master so that master becomes pre-1.1.0 and gets some testing before we decide to release. I incorrectly preferred the wrong version of one or two things while merging, but @gary and a few community members managed to catch them, so I think master is in good shape for pre-1.1.0

  • A fix for V2 slate compatibility with @bdap’s help. This will ensure that post-1.1.0 the foreign api will be able to accept and parse earlier versions of slates, as well as return earlier versions to the caller. I suddenly realized this was missing from 1.1.0 and definitely wanted to get this in before release. It’s also handled much more cleanly than in the V1 api, so there’s a ton of code I look forward to ripping out once we drop the V1 API in favour of V2 and beyond

  • Adding a check_version function to the foreign_api, to let callers know that a wallet is alive and what slates it can accept before trying to send it anything. Should be a help for wallet developers.

  • And the current body of work is on the long awaited Invoiced Transactions feature, which is essentially the inverse of the regular transaction workflow. The payee generates an invoice for a certain amount, adds their output, then sends to the payer to accept it, add their inputs and send back. Will talk about this in more detail once it’s complete, but I’d definitely like to get the API firmed up before 1.1.0 is released.

And as always, other bits and pieces. That’s it for now, and will get back to you next week, hopefully with more info about when 1.1.0 will be shunted into the wild.

Enjoy weekend!

#19

Update Friday, May 3rd 2019

Just a quick update this week as I’m currently travelling over a long weekend

  • Work on the api support for invoice transactions is complete, documented, tested and merged. Complete details for those interested in trying it out via the
    API can get all the documentation via rustdoc

  • I’m currently working on the obvious follow up, enabling invoice transactions in the command-line wallet. Progressing steadily there and should have it ready for 1.1.0 (coming any week now!)

  • As a skunkworks side project/learning experience, I’ve been putting together a rust implementation of SLIP-39: Shamir Secret Sharing for mneumonic codes, which I hope to integrate into the wallet as some point a nice feature-add. Basically, master key sharing where you can take your bip39 mneumonic and split it into a 3 of 5 (for example) threshold scheme to distrubute to family members. The underpinnings here also relate to future multisig wallet features. Much more on this later, but intend to get the implementation public as soon as I’ve got all the embarrassing bits out.

That’s all for now, enjoy long weekend (provided you have one).

4 Likes
#20

Update Friday, May 10th 2019

Good week of steady progress, I think, with demonstrable results for everything mentioned in last week’s hasty post.

Firstly, the command line support for invoice transactions is now completely in place in grin-wallet master. Instructions on how to use it are in the executable help, and writing this has just reminded me that I need to document the new commands on the wiki (unless someone else gets there first). Will get to that shortly.

In slightly bigger news, @quentinlesceller has completed his work migrating CI from travis to Azure pipelines, which means we’re now able to produce builds for grin and grin-wallet for all 3 major platforms with a single tag. Since I’m pretty much done with feature adds for 1.1.0 wallet, I went ahead and did the 1.1.0-beta-2 build, which you can now download and start playing with. However, I don’t believe a 1.1.0-beta.2 build has been created for grin server, so you’ll either have to wait for that or compile grin from master. I’d imagine the grin 1.1.0-beta.2 should be out any day now (going to check on gitter right now)

Also, as mentioned last week, I’ve been working on an implementation of SLIP-39: Shamir secret sharing for mnemonic codes, which I’ve now made public on github at https://github.com/yeastplume/rust-sssmc39. I’ve been working on this as a bit of a side project in the background and hope to get it up a decent enough standard for inclusion in grin’s wallet. The process of doing this has given me a good education in secret sharing, which leads into future research on supporting multisig in the wallet since threshold signatures work using the exact same concepts. The wallet also gets a nice little feature add which I don’t think exists on any other wallet at present. Here’s a small example, from the lib’s test output:

My 12 Word wallet master key (BIP39 mnemonic) is run through the lib and split into a 3 of 5 scheme. The output is as follows:

Group 1 of 1 - 3 of 5 shares required:
pajamas walnut academic acne already upstairs perfect soldier stay trial always cradle midst dryer debut desktop snapshot kernel belong dramatic
pajamas walnut academic agree desktop lobe breathe avoid mule rapids injury task obesity briefing heat born level making medical painting
pajamas walnut academic amazing apart exhaust tenant away yelp mayor blessing prisoner plunge pants very calcium credit unfair rainbow negative
pajamas walnut academic arcade early disease fragment slow pulse software guilt flea paper lizard profile dynamic isolate calcium course agency
pajamas walnut academic axle deliver scene health afraid recall hormone lawsuit kernel cards holy fatigue cradle sympathy award much force

I can then take these split shares and distribute them to 5 family members. Each share is useless on its own, but when I get hit by a bus tomorrow (“he should have quit drinking vodka before noon”,) then any 3 of my 5 family members can get together, recreate my master key and enjoy my grins. There are multiple levels to the scheme, so it’s also possible to create splits that divide your key among multiple groups, like '2 of my personal shares can recreate the seed, or one of my shares plus 3 of 5 family members if I lose a share). There are literally infinite possibilities here, and I hope this implementation is a step toward making key management more… manageable.

So that’s it for now. I’m gonna split now (ROFLLMAOLOLLOLetc…). enjoy the weekend!

7 Likes
#21

Great update, thanks. I know this is probably the wrong place to ask, but is 1.1.0 planned to include Windows binaries?

#22

Great Job Specially Lib and Split one :+1:

#23

Hi Chronos, as you can already see here: https://github.com/mimblewimble/grin-wallet/releases. With the addition of the new CI we are able to produce macOS/Linux/Windows binaries, so yes 1.1.0 will have Windows binaries.

1 Like
#24

Update Friday, May 17th 2019

Got very sucked into completing tests and puttin finishing touches on the Shamir Mnemonics code this week, and happy to say it’s fairly close to completion, with the major remaining tasks being completing some integration testing, CI and release work, and of course whatever reviews we can get on it.

That little segue mostly aside, grin-wallet 1.1.0-beta.3 is released alongside Grin 1.1.0 beta 2., and I hope this is a short lived beta so we can move on to creating a 2.0.0 branch and get started on some major wallet work in advance of the scheduled hard fork.

No firm commits here as to what will get into what version, but as far as major near-term work I’m considering:

  • @jaspervdm’s updated recovery scheme, which should allow for more flexibility with structing wallets (which I still need to review in detail (sorry @jaspervdm).

  • @mcdallas has been looking into encoding slates into minimized base-64, which many people think is worth including.

  • Complete lifecycle management via the API, meaning that you should be able to create and manage multiple wallets via the owner api. Naturally there are a lot of other small little issues that need to be dealt with here, particularly around master key management. This will probably be a meta issue under which some discussion and refactoring work will fall in order to allow it

  • Pulling the BIP-32 code out of grin and place it into a separate repository (as a) it really doesn’t belong in grin and b) and I’d like it split out to be included by the wallet API as part of lifecycle management work above to give greater flexibility in generating keys)

  • Integrating sssmc39 as an option in the wallet api

That’s it for now. Enjoy weekend and hope we can get our final 1.1.0 releases early next week!

6 Likes