Reuse identities across tests

[victorb]

From IRC #ipfs-dev:

20:08:39 <@victorbjelkholm> btw, quick idea, could we ship ipfsd-ctl with a couple of node identities? Like ten or so, would speed up tests if we could use them as a pool of identities and generate new ones when we have tests that uses more than X different nodes
20:08:58 <@victorbjelkholm> instead of generating a ID everytime we create a new node, is a bit wasteful
20:13:43 <dryajov> yeah, I though about that too, I think we need that
20:14:26 <dryajov> not sure if we should ship with identities by default, but maybe structure our tests so that we precreate repos that we need, and then simply reuse them across the tests...
20:15:11 <dryajov> we can have a handy method like `createRepo` that will init it one for us and return it’s location, then we can simply pass it to spawn...
20:16:46 <dryajov> also, setting the key size to 1024 has a marginal effect on overall performance, it only speeds up those tests ~170 of them, on ipfsd-ctl, by about 1 minute...
20:18:06 <dryajov> the bulk of our perf issues, as I mentioned, is in setting config options one by one, after the repo has been initialized… it specially slow for jsipfs, but even then, its just super ineficient
20:19:31 <@victorbjelkholm> dryajov: yeah, figure out the number of nodes needed for the test, generate that many IDs and reuse them. Startup would be slow though
20:19:58 <@victorbjelkholm> think we need (if we go this route) to have them hardcoded, actual files in scm
20:20:12 <@victorbjelkholm> at least a couple
20:20:23 <dryajov> yeah,
20:21:52 <dryajov> victorbjelkholm: I think most tests would be two-three repos… so if we can have a `before` step that would set one for go, one for js, we can then keep reusing them, I think startup is actually gonna go down, since now we pretty much do that for each set of tests...
20:22:57 <dryajov> ah… one issue I see with precreating is, that once we touch one of those repos its tainted...
20:23:01 <@victorbjelkholm> dryajov: yeah, we would need the same amount of keys as the test with the most simultaneous nodes at the same time 
20:23:07 <@victorbjelkholm> it would be fs-repos, it would just be the keys
20:23:17 <@victorbjelkholm> don't think generating the repo takes long time, is the key generation
20:23:37 <dryajov> victorbjelkholm: yeah, I think you’re right, just have the needed identities for our tests, and them copy them for each run… so that we don’t trip over oursefs

TLDR: We can generate X node identities, save them and have them ready to be used for tests so nodes can reuse identities instead of generating a new one (per node) for each test.

[mkg20001]

I've made https://github.com/mkg20001/aegir-keycache which hopefully will fix some of the slowness caused by key generation (demo)

(c&p from #12 (comment))

[victorb]

Not sure having a http endpoint is the right way, I want to avoid having to deal with ports and connections as much as possible. Also, more network requests = slower tests. I would like to see it as a module that you can point to a directory which keeps keys there. Directory structure would basically be directory name is the peer ID and the private key is a file inside that directory.

[mkg20001]

@victorbjelkholm That would be all nice and good weren't it for the browsers.
Browsers don't have read-write access to the filesystem, which is required to store the keys.

[victorb]

@mkg20001 the writes of keys would just happen once and be commited into SCM. Then later tests would just read the keys.

[Kubuxu]

I can't wait for us to move to x25519 crypto. Identity generation problems will be gone then. Private key is 32 bytes of random data and public key is hash and small transformation of that.

---

EDIT: there was a type x29919 -> x25519

[richardschneider]

@Kubuxu Spec for x29919 crypto please! And what is "small transformation"?

[Kubuxu]

@richardschneider see https://ed25519.cr.yp.to/ and https://cr.yp.to/ecdh.html