Grin

Yeastplume - Progress update thread - Oct 18 - Feb 19


#1

October is when the results from the previous funding round kick in, so might as well start a new thread in honour of that fact.

It looks as if T4 is about a week or so away, and most of the work that needs to go in before we can give it a quick pre-deployment test is done. This week was all about getting support for Cuckatoo in place, which meant a lot of updates to grin, grin-miner and cuckoo-miner. More specifically:

  • cuckoo-miner, which is the library that builds cuck(at)oo plugins for inclusion in other projects, was updated with 2 new cuckatoo miners, courtesy of @tromp. There’s a cucakatoo-lean cpu miner, which takes between 25-35 seconds to find a graph on my machine depending on settings, and a likely much-more-useful cuckatoo-gpu miner, which is very comparable to the cuckoo gpu miner at around 4 GPS on a 1080ti. (A mean-cpu miner is forthcoming, which i’d estimate would do a single graph in a few seconds). In addition, the library was updated to allow for variable cuckoo sizes… it should now be possible to mine using whatever size cuckatoo plugin you want (above 29), and have any solutions passed back to Grin along with what size cuck(at)oo graph the solution is for.

*grin-miner was updated to use the new cuckatoo plugins, as well as pass back cuckoo-size data to grin. Also a few enhancements to the TUI while I was there to display the plugin name and graph size for each plugin.

*And finally the Grin PR that integrates all of the PoW work over the past couple of weeks Mostly switching on flags that were already set up, but a few changes there as well to support multiple mining sizes.

Only 2 things left on my list for T4 now are to clean up some automated tests (results of the pow change) as well as integrate the final reviewed Elements bullet-proof code into our secp branch. (Some hard-forking changes went into there recently, and I want to be sure Grin T4 has the latest BP code before it deploys).

Besides that @antioch and @gary have been looking at sync issues on T3, and it looks like they’re making good progress there. Once they’re happy with that, then we should be just about ready to merge this and all of the previous wallet, bulletproof, and aggsig work to create a (hopefully short-lived) T4 branch to test changes before we deploy what should hopefully be the final test network (though one can never say never).

That’s it for now, testnet deployments are always exciting times, and hopefully we’ll be mining T4 grins any day now.


#2

Update Saturday Oct 13th, 2018

We don’t usually have a month going by where someone doesn’t ask this question ‘Why did you implement Grin in Rust instead of C’ in one of our various communication options. Sometimes this is accompanied by a little diatribe in which the poster implies that if only we had implemented Grin in a language that ‘everybody knows like C++’ they would have been able to lend their considerable skills to the project.

Alas, we must forge ahead without these people (I don’t think we’ve heard from any of these people twice). But my answer to anyone who asks either online or in real life is simply ‘When you write code in Rust, it does exactly what you think it’s going to do, every time.’

A lump of C/C++ code of any meaningful size rarely gets itself into a state where it does what’s intended without a considerable amount of time spent in front of a debugger tracing down odd threading issues, investigating segfaults, and dealing with general weirdness. Of course it’s possible to include tools or implement standards to lessen these issues, but it’s time consuming, and every team will implement all of this differently for every project. Then of course different compilers will produce different code and there are approximately 4000 different C++ standards.

Because Grin uses rust, we rarely have to deal with this sort of oddness, which frees up time to focus on the actual issues rather than spending weeks in LLDB. Now there are plenty of article about the merits of Rust on the Internet, and I’m not trying to create another one. This is just for context when I talk about this week’s work.

So Cuckoo-miner, which provides a Rust bridge to slightly-modified versions of @tromp’s cuckoo/cuckatoo miners, is a Rust-C hybrid that I put together well over a year ago when I knew far less about Rust than I do now. The Rust-side is little more than an interface meant to be called by other rust projects, while the C side is an FFI shim to call the C code in the C solvers, however it also contains quite a bit of C code for queue management and to enable the multi threading needed to run multiple devices at once. When I was originally putting this together, I decided to do the queue management on the C side because I reckoned it had to work as fast as possible and doing it in C is the way to do this.

Nowadays, I realize that is entirely the wrong approach to take. When you’re using Rust, any decision to use C should be similar to when you’re primarily using C and are considering doing an algorithm in assembly. You might be able to make it faster, but an optimizing C compiler is generally going to produce far better assembly than you can hand-write, unless it’s something very specific and you really know what you’re doing. Similarly, you should never consider doing something in C when working primarily in Rust for ‘speed issues’ unless there’s some demonstrable need. The cuck(at)oo solvers themselves are good candidates for being written in C as they’re very hardware specific and need to interface into other C frameworks. The queue management portion of cuckoo-miner is not. It should have been kept in Rust as much as possible and the interface into the plugins themselves kept far thinner.

Reason I’m saying this is because over the last two weeks I integrated 3 solvers into cuckoo-miner, (mean CPU, lean CPU and cuda GPU) and each and every time I spent hours tracking down obscure issues that would either have been caught at compile time if those portions of the code had been using Rust, or simply not have been able to happen in the first place. The process of tracking down these issues are extremely frustrating and complicated exercises, usually hours spent hopelessly starting into a debugger with your branch of the code littered with print statements. This kind of session does not happen when working on Grin itself, and I don’t think I’d ever advocate C for a new project unless it’s absolutely required. Rust would simply a better choice in 99% of cases.

So with that little glorified tweet out of the way, here’s what I actually did this week:

  • Integrated the Mean CPU miner into Grin-miner, which took a bit longer than expected as explained above. I’d love to re-do parts of cuckoo miner to move the queue management into Rust, but it works now, isn’t mission-critical and can be rewritten at any time. Probably more important things to focus on for the time being.

  • Updated libsecp with the very latest and hopefully final version of element’s bulletproofs A few critical (but T3 consensus breaking) fixes in there, so glad to get it in before T4. Also a few upstream changes to how commits are represented meant a bit more headache integrating this than anticipated (seems to have been the theme of the week)

  • Creation of the testnet4 branch and integration of all of the wallet, aggsig, bulletproof PRs, etc. I’ll go into far more detail on T4 next week (with an exact list of what’s changed), as there are still a few bits and pieces on it being done, suffice it to say this is where all the focus is now.

  • Finally a bit of a breather between large coding efforts to catch up and better understand some recent concepts, particularly Header MMRs, the newly proposed prev_root field and flyclient, our dual PoW difficulty adjustments and a couple of other things. I can get a bit of myopia sometimes when i’m deep in my own things, so find it helpful to come up for air once in a while and get back up to speed with everything else that’s going on (at the rate things are moving at the moment that could be a full-time job in itself).

That’s it for now, and the next week will most likely be about pre-release T4 testing and addressing whatever bits and pieces come up there. Everything is almost in place, and looking very forward to getting T4 out there.


#3

Update Friday, Oct 19th 2018

Going to be very quick this week as just about everything this week was to do with the launch of testnet4. No large-ish chunks of work to note, because most of the week was spent testing, fiddling around, committing fixes, merging, committing fixes to fixes, making/cleaning up messes, and all of the other things you do when you’re trying to release a new iteration of something. Though T4 is probably known far and wide now, a list of changes is in the announcement link above.

All of the fiddling about with mining code has made me realise I very much do want to clean up/rewrite most of cuckoo-miner before mainnet, especially given all the PoW tweaking that still needs to happen. Will be turning my attention to that next, and would like to get cuckoo-miner to a state where it launches the minimal number of threads required and needs minimal tweaking of @tromp’s solvers upstream to get them integrated into Grin-miner when they change. This should lessen the instances of odd segfaulting behaviour, and hopefully make it far easier to track down issues when they do occur.

But for now, T4 out, it’s Friday, and it’s time for a celebration. I dedicate this particular vodka-night to T4.


#4

Update Friday, October 26th 2018

As indicated last week shortly before the celebratory vodka-night, I had a good time this week bringing grin-miner, to a more refined standard. The original version was put together by someone who didn’t have a lot of experience with Rust (i.e. me, about a year and a half ago,) was overly complex and contained far more custom C than necessary. The new version has been pared down and moves all thread control Rust-side, eliminating the potential for 95% of the types of bugs I used to need to track down every time something went wrong in the old version. The new version is a much safer and stable experience, it’s eliminated several odd bugs already and has been very stable running multiple cuda devices on the yeastmonster. So a week spent here should save months down the line.

In detail, the work involved:

  • Instead of keeping a custom fork of the solvers, with @tromp’s assistance I made quite a few changes to the main cuckoo/cuckatoo repository to enable linkage into other projects while retaining the ability to run them via command line. This means that going forward, there will be no extra layers added to the solvers when they’re run in grin-miner, and they should work exactly as they do from the command line. This means changes/updates to solvers can be instantly picked up in grin-miner, and, so long as future solvers are written to the same interface, they should be easy to integrate.

  • Reducing the plugin interface to 4 functions (whereas it had about a dozen before, most of them adding no real value)

  • Instead of the 3-repository layout that grew between our custom cuckoo fork, the separate cuckoo-miner library and grin-miner, all code is now in the grin-miner project which just pulls https://github.com/tromp/cuckoo as a sub module

  • Chopping/simplifying much of the mining control, removing unnecessary function calls and eliminating threading in favour of rust mpsc message queues where it made sense. The main control portion of the cuckoo-miner lib itself is now just a few hundred lines of rust (as opposed to being split between rust and C): https://github.com/mimblewimble/grin-miner/blob/master/cuckoo-miner/src/miner/miner.rs

  • Simplifying the configuration options in grin-miner.toml. It should be much easier and clearer how to set up plugins for multiple devices now.

  • Many QOL bits and pieces in grin-miner, most noticeably more graceful error handling when solvers are misconfigured or throw errors.

Still one or two small things I’d like to do on grin-miner, but they can come over time. The biggest outstanding issue is that the solvers aren’t currently respecting a flag that asks them to stop solving and return immediately, which would be a bit of an issue on slower solvers meaning it still takes them several seconds to switch jobs once notified of a new block. This might lead to some rejected solutions, but those should be cleaned up once @tromp has done a bit of work on getting solvers to exit on request.

So I’m happy with progress this week, and with this new version it’s probably a better time than ever to start test-mining Grin. Will need to go through a few old threads and update them, but for now the build instructions for grin-miner itself combined with the documentation in grin-miner.toml should be enough to get most people going.

Maybe another celebratory vodka-night is in order, who knows? Have a good weekend all!


#5

Thanks for the terrific job you’re doing with grin-miner!

Is it at all possible to add to the tui the chance that a machine has at mining a block based on its current and the network’s hash rates? Something like: “Mining Odds: 0.7%”


#6

never been so enamoured with yeast in any form. I even prefer plumes over bread now.


#7

Update Friday Nov 2nd, 2018

Bit all over the place this week, as it was shortened due to bank+school holidays (i.e. everyone else gets a holiday while I have to contain Yeastlings,) and also filled with an urgent need to get two coherent and non-embarrassing presentations together in advance of grincon0.

So on the presentation front, got the wallet one mostly done, and at least have an outline for the general Grin Intro. The latter is far more difficult given that it’s the first fully-human event that’ll happen at Grincon and there’s no real precedent here. I have images in my head of trying to whip up the crowd into a Grin-Euphoric frenzy, but I don’t trust my oration skills enough for that, so will probably go with something a bit more subdued. Whatever I end up coming with, it’s bloody time-consuming. Development work has suffered as a result, but given this is the first grincon and it’ll be live-streamed all over and recorded for posterity, I’ll take the liberty of wearing a marketing hat for a couple of days. This stuff is probably important too. I hope to get most of it out of the way over the weekend so the other speakers can review a bit, then have a clearer mind for deeper tech work next week.

Code wise, I did manage to get some fixes to wallet-restore in that I’d noticed a while ago but never got around to looking at while focused on the miner, and I’m also very aware of a few wallet issues that have popped up (I’ve discovered a few myself,) that I’ll attend to when next I sit down in front of vim/tmux. Also, the next major chunk of work I’m going to attend to will likely be to make a proper thing out of the grin web wallet. I resurrected it locally to look at a few issues preventing it from working right this moment, but more importantly we had a few good conversations this week about how we’re going to approach it.

We’re going to spec it out some, get the user stories together until we have a proper brief, and then we’re going to go about getting it designed properly. This is a really, really good time for any designers in the community to get properly involved in turning the web-wallet into as good a user experience as possible, so if you’ve been thinking about adding your talents in that way, please do step forward. We have a plan to handle the design work if nobody does come forward, but we’re holding out hope for having the design community-driven.

If you have strong opinions about how the web wallet should work, the issue for user stories is here:

We had a big talk about it, @lehnberg graciously took the action of putting up the issue, and I’ve added nothing to it since. (I’m getting to that as well, just as soon as the grincon prep is under control.)

Right, back to google slides. I’m going to spend the weekend ignoring the family and gesticulating madly while pretending to address a crowd in my head.


#8

Hey, happy to help design here. At the very least I can put together some UX flows.

Has anyone written a spec for this?


#9

Speaking of Rust, I’m going to dive into learning Rust. Are there any resources that were particularly helpful that you’d recommend? I am aware of The Rust Programming Language book. Thanks.


#10

Also happy to assist with any UX / UI needs.


#11

Im a UI/UX designer and id love to lend a hand in whatever way you see fit.


#12

@birdonawire @darsmith @permaetheus that’s fantastic! We’d love to have your help. As Yeast mentioned below we’re trying to figure out user stories / use cases and we’ll have this drive a brief for a front end experience. Once there’s such a brief I’ll be sure to reach out to you all (pm me your emails if you prefer otherwise we’ll just post to here / GitHub).

In the meanwhile, a good way to help before the brief would be to work with us on the user stories. :v:


#13

I can help with the UX design too. Please, count me in. Later, will post some relevant use cases in Github.


#14

Update Monday November 12th, 2018

Missed out on filling in this update on Friday, but I think, just for this week’s entry, I’ll claim that I’ve already done my part in keeping the community abreast of current happenings :smiley:

For posterity, Grincon0 was last Friday evening. Was a great time had by all, and playlist covering the entire thing can be found right here!

Will be back to regularly scheduled proper updates at the end of this week.


#15

Update Friday Nov 16, 2018

Back to wallet concerns, and @mcdallas came up with an interesting proof of concept that you can take a further look at here: https://github.com/mcdallas/gringotts-rust. Basically, it’s a shim between keybase and the wallet listener that lets you exchange Slate data via keybase. Everyone on the team seemed to think this was a great idea in principle, and the conversation extended to the notion of having a common API in place to allow the community to experiment with different slate-exchange solutions .

So to this end, I’ve decided another round of refactoring on the Wallet API is in order, this time to more strongly decouple transaction operations from the actual exchange. So to this end I’m working on another large-ish PR: https://github.com/mimblewimble/grin/pull/1983

It’s work in progress so far, but I think it’s headed in the right direction. The wallet API was a bit cluttered with mostly redundant functions that did things like ‘build a transaction and send via http’ or ‘build a transaction and save to a file’. I’ve changed that to make the API a bit more granular, so now there’s a single ‘build a transaction’ function that just returns the slate to the caller and lets the caller deal with how to send it around. The caller is responsible for doing a bit more under this design, but it’s ultimately much more flexible.

As for sending and receiving, I’m trying to create a common interface to allow authors to implement different communication methods that the command-line client should just be able to pick up and use without further changes to the code. You can see here what it looks like at the moment, but keep in mind it’s preliminary and subject to change. At the moment I have an HTTP Adapter that handles sending and listening as per usual, but I’ve also implemented a File Adapter that now takes care of loading files into slates and saving them out again. As a result, the main wallet code looks exactly the same no matter what communication method is used, the only thing that differs is the type of Adapter being used. If that works out, hopefully we can test the architecture a bit more fully by integrating a keybase adapter.

So deep back into code this week after last week’s Extroversion. Plan is to just hide here for a bit and try and shape the wallet api into its final(ish) form and hammer out any remaining annoyances.

Take care, and enjoy weekend!