Yeastplume - Progress update thread - Oct 18 - Feb 19


#1

October is when the results from the previous funding round kick in, so might as well start a new thread in honour of that fact.

It looks as if T4 is about a week or so away, and most of the work that needs to go in before we can give it a quick pre-deployment test is done. This week was all about getting support for Cuckatoo in place, which meant a lot of updates to grin, grin-miner and cuckoo-miner. More specifically:

  • cuckoo-miner, which is the library that builds cuck(at)oo plugins for inclusion in other projects, was updated with 2 new cuckatoo miners, courtesy of @tromp. There’s a cucakatoo-lean cpu miner, which takes between 25-35 seconds to find a graph on my machine depending on settings, and a likely much-more-useful cuckatoo-gpu miner, which is very comparable to the cuckoo gpu miner at around 4 GPS on a 1080ti. (A mean-cpu miner is forthcoming, which i’d estimate would do a single graph in a few seconds). In addition, the library was updated to allow for variable cuckoo sizes… it should now be possible to mine using whatever size cuckatoo plugin you want (above 29), and have any solutions passed back to Grin along with what size cuck(at)oo graph the solution is for.

*grin-miner was updated to use the new cuckatoo plugins, as well as pass back cuckoo-size data to grin. Also a few enhancements to the TUI while I was there to display the plugin name and graph size for each plugin.

*And finally the Grin PR that integrates all of the PoW work over the past couple of weeks Mostly switching on flags that were already set up, but a few changes there as well to support multiple mining sizes.

Only 2 things left on my list for T4 now are to clean up some automated tests (results of the pow change) as well as integrate the final reviewed Elements bullet-proof code into our secp branch. (Some hard-forking changes went into there recently, and I want to be sure Grin T4 has the latest BP code before it deploys).

Besides that @antioch and @gary have been looking at sync issues on T3, and it looks like they’re making good progress there. Once they’re happy with that, then we should be just about ready to merge this and all of the previous wallet, bulletproof, and aggsig work to create a (hopefully short-lived) T4 branch to test changes before we deploy what should hopefully be the final test network (though one can never say never).

That’s it for now, testnet deployments are always exciting times, and hopefully we’ll be mining T4 grins any day now.


#2

Update Saturday Oct 13th, 2018

We don’t usually have a month going by where someone doesn’t ask this question ‘Why did you implement Grin in Rust instead of C’ in one of our various communication options. Sometimes this is accompanied by a little diatribe in which the poster implies that if only we had implemented Grin in a language that ‘everybody knows like C++’ they would have been able to lend their considerable skills to the project.

Alas, we must forge ahead without these people (I don’t think we’ve heard from any of these people twice). But my answer to anyone who asks either online or in real life is simply ‘When you write code in Rust, it does exactly what you think it’s going to do, every time.’

A lump of C/C++ code of any meaningful size rarely gets itself into a state where it does what’s intended without a considerable amount of time spent in front of a debugger tracing down odd threading issues, investigating segfaults, and dealing with general weirdness. Of course it’s possible to include tools or implement standards to lessen these issues, but it’s time consuming, and every team will implement all of this differently for every project. Then of course different compilers will produce different code and there are approximately 4000 different C++ standards.

Because Grin uses rust, we rarely have to deal with this sort of oddness, which frees up time to focus on the actual issues rather than spending weeks in LLDB. Now there are plenty of article about the merits of Rust on the Internet, and I’m not trying to create another one. This is just for context when I talk about this week’s work.

So Cuckoo-miner, which provides a Rust bridge to slightly-modified versions of @tromp’s cuckoo/cuckatoo miners, is a Rust-C hybrid that I put together well over a year ago when I knew far less about Rust than I do now. The Rust-side is little more than an interface meant to be called by other rust projects, while the C side is an FFI shim to call the C code in the C solvers, however it also contains quite a bit of C code for queue management and to enable the multi threading needed to run multiple devices at once. When I was originally putting this together, I decided to do the queue management on the C side because I reckoned it had to work as fast as possible and doing it in C is the way to do this.

Nowadays, I realize that is entirely the wrong approach to take. When you’re using Rust, any decision to use C should be similar to when you’re primarily using C and are considering doing an algorithm in assembly. You might be able to make it faster, but an optimizing C compiler is generally going to produce far better assembly than you can hand-write, unless it’s something very specific and you really know what you’re doing. Similarly, you should never consider doing something in C when working primarily in Rust for ‘speed issues’ unless there’s some demonstrable need. The cuck(at)oo solvers themselves are good candidates for being written in C as they’re very hardware specific and need to interface into other C frameworks. The queue management portion of cuckoo-miner is not. It should have been kept in Rust as much as possible and the interface into the plugins themselves kept far thinner.

Reason I’m saying this is because over the last two weeks I integrated 3 solvers into cuckoo-miner, (mean CPU, lean CPU and cuda GPU) and each and every time I spent hours tracking down obscure issues that would either have been caught at compile time if those portions of the code had been using Rust, or simply not have been able to happen in the first place. The process of tracking down these issues are extremely frustrating and complicated exercises, usually hours spent hopelessly starting into a debugger with your branch of the code littered with print statements. This kind of session does not happen when working on Grin itself, and I don’t think I’d ever advocate C for a new project unless it’s absolutely required. Rust would simply a better choice in 99% of cases.

So with that little glorified tweet out of the way, here’s what I actually did this week:

  • Integrated the Mean CPU miner into Grin-miner, which took a bit longer than expected as explained above. I’d love to re-do parts of cuckoo miner to move the queue management into Rust, but it works now, isn’t mission-critical and can be rewritten at any time. Probably more important things to focus on for the time being.

  • Updated libsecp with the very latest and hopefully final version of element’s bulletproofs A few critical (but T3 consensus breaking) fixes in there, so glad to get it in before T4. Also a few upstream changes to how commits are represented meant a bit more headache integrating this than anticipated (seems to have been the theme of the week)

  • Creation of the testnet4 branch and integration of all of the wallet, aggsig, bulletproof PRs, etc. I’ll go into far more detail on T4 next week (with an exact list of what’s changed), as there are still a few bits and pieces on it being done, suffice it to say this is where all the focus is now.

  • Finally a bit of a breather between large coding efforts to catch up and better understand some recent concepts, particularly Header MMRs, the newly proposed prev_root field and flyclient, our dual PoW difficulty adjustments and a couple of other things. I can get a bit of myopia sometimes when i’m deep in my own things, so find it helpful to come up for air once in a while and get back up to speed with everything else that’s going on (at the rate things are moving at the moment that could be a full-time job in itself).

That’s it for now, and the next week will most likely be about pre-release T4 testing and addressing whatever bits and pieces come up there. Everything is almost in place, and looking very forward to getting T4 out there.


#3

Update Friday, Oct 19th 2018

Going to be very quick this week as just about everything this week was to do with the launch of testnet4. No large-ish chunks of work to note, because most of the week was spent testing, fiddling around, committing fixes, merging, committing fixes to fixes, making/cleaning up messes, and all of the other things you do when you’re trying to release a new iteration of something. Though T4 is probably known far and wide now, a list of changes is in the announcement link above.

All of the fiddling about with mining code has made me realise I very much do want to clean up/rewrite most of cuckoo-miner before mainnet, especially given all the PoW tweaking that still needs to happen. Will be turning my attention to that next, and would like to get cuckoo-miner to a state where it launches the minimal number of threads required and needs minimal tweaking of @tromp’s solvers upstream to get them integrated into Grin-miner when they change. This should lessen the instances of odd segfaulting behaviour, and hopefully make it far easier to track down issues when they do occur.

But for now, T4 out, it’s Friday, and it’s time for a celebration. I dedicate this particular vodka-night to T4.


#4

Update Friday, October 26th 2018

As indicated last week shortly before the celebratory vodka-night, I had a good time this week bringing grin-miner, to a more refined standard. The original version was put together by someone who didn’t have a lot of experience with Rust (i.e. me, about a year and a half ago,) was overly complex and contained far more custom C than necessary. The new version has been pared down and moves all thread control Rust-side, eliminating the potential for 95% of the types of bugs I used to need to track down every time something went wrong in the old version. The new version is a much safer and stable experience, it’s eliminated several odd bugs already and has been very stable running multiple cuda devices on the yeastmonster. So a week spent here should save months down the line.

In detail, the work involved:

  • Instead of keeping a custom fork of the solvers, with @tromp’s assistance I made quite a few changes to the main cuckoo/cuckatoo repository to enable linkage into other projects while retaining the ability to run them via command line. This means that going forward, there will be no extra layers added to the solvers when they’re run in grin-miner, and they should work exactly as they do from the command line. This means changes/updates to solvers can be instantly picked up in grin-miner, and, so long as future solvers are written to the same interface, they should be easy to integrate.

  • Reducing the plugin interface to 4 functions (whereas it had about a dozen before, most of them adding no real value)

  • Instead of the 3-repository layout that grew between our custom cuckoo fork, the separate cuckoo-miner library and grin-miner, all code is now in the grin-miner project which just pulls https://github.com/tromp/cuckoo as a sub module

  • Chopping/simplifying much of the mining control, removing unnecessary function calls and eliminating threading in favour of rust mpsc message queues where it made sense. The main control portion of the cuckoo-miner lib itself is now just a few hundred lines of rust (as opposed to being split between rust and C): https://github.com/mimblewimble/grin-miner/blob/master/cuckoo-miner/src/miner/miner.rs

  • Simplifying the configuration options in grin-miner.toml. It should be much easier and clearer how to set up plugins for multiple devices now.

  • Many QOL bits and pieces in grin-miner, most noticeably more graceful error handling when solvers are misconfigured or throw errors.

Still one or two small things I’d like to do on grin-miner, but they can come over time. The biggest outstanding issue is that the solvers aren’t currently respecting a flag that asks them to stop solving and return immediately, which would be a bit of an issue on slower solvers meaning it still takes them several seconds to switch jobs once notified of a new block. This might lead to some rejected solutions, but those should be cleaned up once @tromp has done a bit of work on getting solvers to exit on request.

So I’m happy with progress this week, and with this new version it’s probably a better time than ever to start test-mining Grin. Will need to go through a few old threads and update them, but for now the build instructions for grin-miner itself combined with the documentation in grin-miner.toml should be enough to get most people going.

Maybe another celebratory vodka-night is in order, who knows? Have a good weekend all!


#5

Thanks for the terrific job you’re doing with grin-miner!

Is it at all possible to add to the tui the chance that a machine has at mining a block based on its current and the network’s hash rates? Something like: “Mining Odds: 0.7%”


#6

never been so enamoured with yeast in any form. I even prefer plumes over bread now.


#7

Update Friday Nov 2nd, 2018

Bit all over the place this week, as it was shortened due to bank+school holidays (i.e. everyone else gets a holiday while I have to contain Yeastlings,) and also filled with an urgent need to get two coherent and non-embarrassing presentations together in advance of grincon0.

So on the presentation front, got the wallet one mostly done, and at least have an outline for the general Grin Intro. The latter is far more difficult given that it’s the first fully-human event that’ll happen at Grincon and there’s no real precedent here. I have images in my head of trying to whip up the crowd into a Grin-Euphoric frenzy, but I don’t trust my oration skills enough for that, so will probably go with something a bit more subdued. Whatever I end up coming with, it’s bloody time-consuming. Development work has suffered as a result, but given this is the first grincon and it’ll be live-streamed all over and recorded for posterity, I’ll take the liberty of wearing a marketing hat for a couple of days. This stuff is probably important too. I hope to get most of it out of the way over the weekend so the other speakers can review a bit, then have a clearer mind for deeper tech work next week.

Code wise, I did manage to get some fixes to wallet-restore in that I’d noticed a while ago but never got around to looking at while focused on the miner, and I’m also very aware of a few wallet issues that have popped up (I’ve discovered a few myself,) that I’ll attend to when next I sit down in front of vim/tmux. Also, the next major chunk of work I’m going to attend to will likely be to make a proper thing out of the grin web wallet. I resurrected it locally to look at a few issues preventing it from working right this moment, but more importantly we had a few good conversations this week about how we’re going to approach it.

We’re going to spec it out some, get the user stories together until we have a proper brief, and then we’re going to go about getting it designed properly. This is a really, really good time for any designers in the community to get properly involved in turning the web-wallet into as good a user experience as possible, so if you’ve been thinking about adding your talents in that way, please do step forward. We have a plan to handle the design work if nobody does come forward, but we’re holding out hope for having the design community-driven.

If you have strong opinions about how the web wallet should work, the issue for user stories is here:

We had a big talk about it, @lehnberg graciously took the action of putting up the issue, and I’ve added nothing to it since. (I’m getting to that as well, just as soon as the grincon prep is under control.)

Right, back to google slides. I’m going to spend the weekend ignoring the family and gesticulating madly while pretending to address a crowd in my head.


#8

Hey, happy to help design here. At the very least I can put together some UX flows.

Has anyone written a spec for this?


#9

Speaking of Rust, I’m going to dive into learning Rust. Are there any resources that were particularly helpful that you’d recommend? I am aware of The Rust Programming Language book. Thanks.


#10

Also happy to assist with any UX / UI needs.


#11

Im a UI/UX designer and id love to lend a hand in whatever way you see fit.


#12

@birdonawire @darsmith @permaetheus that’s fantastic! We’d love to have your help. As Yeast mentioned below we’re trying to figure out user stories / use cases and we’ll have this drive a brief for a front end experience. Once there’s such a brief I’ll be sure to reach out to you all (pm me your emails if you prefer otherwise we’ll just post to here / GitHub).

In the meanwhile, a good way to help before the brief would be to work with us on the user stories. :v:


#13

I can help with the UX design too. Please, count me in. Later, will post some relevant use cases in Github.


#14

Update Monday November 12th, 2018

Missed out on filling in this update on Friday, but I think, just for this week’s entry, I’ll claim that I’ve already done my part in keeping the community abreast of current happenings :smiley:

For posterity, Grincon0 was last Friday evening. Was a great time had by all, and playlist covering the entire thing can be found right here!

Will be back to regularly scheduled proper updates at the end of this week.


#15

Update Friday Nov 16, 2018

Back to wallet concerns, and @mcdallas came up with an interesting proof of concept that you can take a further look at here: https://github.com/mcdallas/gringotts-rust. Basically, it’s a shim between keybase and the wallet listener that lets you exchange Slate data via keybase. Everyone on the team seemed to think this was a great idea in principle, and the conversation extended to the notion of having a common API in place to allow the community to experiment with different slate-exchange solutions .

So to this end, I’ve decided another round of refactoring on the Wallet API is in order, this time to more strongly decouple transaction operations from the actual exchange. So to this end I’m working on another large-ish PR: https://github.com/mimblewimble/grin/pull/1983

It’s work in progress so far, but I think it’s headed in the right direction. The wallet API was a bit cluttered with mostly redundant functions that did things like ‘build a transaction and send via http’ or ‘build a transaction and save to a file’. I’ve changed that to make the API a bit more granular, so now there’s a single ‘build a transaction’ function that just returns the slate to the caller and lets the caller deal with how to send it around. The caller is responsible for doing a bit more under this design, but it’s ultimately much more flexible.

As for sending and receiving, I’m trying to create a common interface to allow authors to implement different communication methods that the command-line client should just be able to pick up and use without further changes to the code. You can see here what it looks like at the moment, but keep in mind it’s preliminary and subject to change. At the moment I have an HTTP Adapter that handles sending and listening as per usual, but I’ve also implemented a File Adapter that now takes care of loading files into slates and saving them out again. As a result, the main wallet code looks exactly the same no matter what communication method is used, the only thing that differs is the type of Adapter being used. If that works out, hopefully we can test the architecture a bit more fully by integrating a keybase adapter.

So deep back into code this week after last week’s Extroversion. Plan is to just hide here for a bit and try and shape the wallet api into its final(ish) form and hammer out any remaining annoyances.

Take care, and enjoy weekend!


#16

Update Friday November 23rd, 2018

There appears to be some sort of mandatory thanking-type holiday going on in the US, so the community’s been a bit more quiet than usual over the past couple of days. I’ve had a solid week smashing away at the keyboard in front of several vim/tmux windows. The small matter of a release date being announced (on or around Jan 15th) has helped focus the mind on what absolutely, positively needs to be done by then, so I’ve altered course slightly. Here’s what happened:

  • The ‘plugin’ refactoring PR referenced last week is done, tested, and merged. There should still be a few things left to work out on that functionality, such as how to pass parameters into a plugin and whether/how to separate ‘plugins’ builds from the main code base. I’m hoping someone from the community interested in seeing wallet exchanges happen over their medium of choice will have a look at it, but otherwise I’m quite interested in integrating a keybase exchange when it can be appropriately prioritised.

  • Of more pressing concern now is BIP39 support and better handling of the wallet seed BIP 39 simply means “wallet recovery word lists”, but some changes to how we use the password and store seeds were also necessary to support this. This, I think, was the most obvious piece of user functionality missing from the command-line wallet, and will be in place once a cursory review or two is done and I push the big green merge button. Full details on what’s changing exactly are in the comments section of the PR, and the ever-growing wallet usage doc has been updated to reflect the changes.

  • I’ve also undertaken to start properly documenting all of the wallet API layers as an ongoing project. Putting together documentation, especially good documentation with code examples is quite painful (rust even tests documentation code examples to make sure they compile,) and much better done over time in short, focused sessions as opposed to marathons where you start taking shortcuts cause you’re sick of looking at it after 10 minutes. So more to come there over time, particularly on the more ‘inner’ layers of the wallet libraries, which I think are less likely to change now.

Just quickly on that, while the lower layer apis (libtx and libwallet) are of decent enough quality now (or so I think), the wrapper portion that currently comprises the HTTP owner api listener has more or less been thrown together over time and there’s no coherent JSON api into the various functions. This will need to be addressed at some stage, but it’ll probably be on lower priority than ensuring the command line wallet is ready for launch. I know quite a few people have been using it for their own projects, so if you have any suggestions for making the HTTP wrapper more coherent, let me know. Also keep in mind that when this happens everyone will likely need to update their applications to use a newer API.

Right, that’s it for now. Please return to your thankful bird slaughter or further fraying your relationships with your family or whatever it is you do over there.


#17

Update Friday Nov 30th 2018.

Nothing like a release date to sharpen the senses, especially one with tons of Yuletide disruption between now and it.

The wallet march continues:

  • Testing and bugfixes based on user findings in the field. With release date looming, and most of the functionality complete, I’ve started to pay more a bit more attention to these.

  • Endless documentation of the wallet APIs. Rustdoc is great, but it’s very time consuming to put together, especially given that it goes as far as compiling your code examples to make sure they’re current. I have about half of the OwnerAPI documented, with the rest coming very soon (I think I have most of the awkward ones done. Painful but worth taking time on.

  • Various API refinements springing from above. As I’m documenting, I come across things in the API and say to myself “this is pants” and remove or rework the offending API call rather than document it. (Actually takes less time in most cases).

  • Addition of an optional Slate message, basically provide a space for a user to include a signed message in the slate (should hopefully enhance UX somewhere down the line.

  • Also starting to do some thinking about what the HTTP/JSON api should actually look like. Thoughts thus far are in the issue, so won’t repeat them here.

  • Also noteworthy: @mcdallas is implementing a keybase plugin in accordance with the plugin work last week, and it looks really promising, very excited to see this.

Coming up: more documentation, perhaps a bit of a rework of the wallet command-line code (Igno pointed out that file was getting large, and he’s correct,) some grin-miner modifications to assist plugin developers (@hashmap is in the middle of one), more thought about how to build out the V2 plugin API, and bug fixes, performance testing,… and that doesn’t touch the backlog of ‘must-have’ issues that we have marked for release.

…and all of this with holiday disruption season coming up. Fun times ahead! (But don’t think I’d rather be anywhere else.)


#18

Update Monday Dec 10th, 2018

Completely forgot to write anything here with all of the excitement going on, the definition of ‘excitement’ being ‘broken the build during the time I should have been writing this’.

Last week’s work was mostly about re-organising the wallet command line code properly to make it more consistent and manageable, as well as changes to support end-to-end automated testing of the command line client (to the point where tests are actually parsing a command line and executing the result). Want to get moving on this weeks task’s (more wallet fixes, wallet performance, look into these mining issues everyone seems to be having to see if there’s anything in it,) so will keep it short and save a longer update for Friday.

So off to Vim I go…


#19

Update Friday Dec14th, 2018

Bug fixes, testing, stability, issue resolution the past couple of weeks, likely to continue for the next few weeks as we approach launch. Nothing earth shattering here, just many smaller items and fixes as befits the weeks before a release. The kind of development that makes you feel a bit over-excitable and scatter-brained. As a sampling of the kinds of things I’m addressing:

And of course, the large refactor of the wallet command line code and test organisation is complete: https://github.com/mimblewimble/grin/pull/2114

Also a few mining fixes, starting to look at wallet performance, and @jaspervdm is doing some work on converting wallet outputs to switch commitments that I need to be reviewing as well: https://github.com/mimblewimble/grin/pull/2157.

All in all, things are looking relatively stable still quite a few issues left to address as ‘must-haves’ before launch, but nothing too massive. And of course, mainnet release is just another development milestone. Once this round of consolidation and main-net launch excitement is over, we’re going to keep right on with additions and improvements.

Right, expect more of the same next week from everyone, and enjoy weekend. Only 376 mod 365 shopping days until Christmas!