Aether on Java, how to have info on incorrect requests

Bugs & issues
#1

Linux, Java
Aether Community Edition v2.0.0-dev.15, build 2011262249.19338c93

I am trying to do a POST request for /v0/c0/boards

Actual behavior
Response is Server returned HTTP response code: 400 for URL: https://localhost:37107/v0/c0/boards

The logs from Aether are
2021/01/09 10:56:30 BoardsPOST POST request parsing failed. Error: &errors.errorString{s:“The request is syntactically valid JSON, but it does not include certain vital information. Remote: 127.0.0.1:59088”}
, Request Header: http.Header{“Accept”:[]string{“application/json”}, “Connection”:[]string{“keep-alive”}, “Content-Length”:[]string{“689”}, “Content-Type”:[]string{“application/json; utf-8”}, “User-Agent”:[]string{“Java/11.0.9.1”}}
, Request Body: api.ApiResponse{NodeId:"", NodePublicKey:“4139c018f836711e9d9d2790e4f78091e354e061b1a1e2371ac5d8a5200bbd69”, Signature:"", ProofOfWork:“MIM1:21::::xHqgPqcjsKOglbii:362467:”, Nonce:“6280b90da285a9d5ab51316287684e3e170b9e1cfba0ae756495fbae54e97ee1”, EntityVersion:1, Address:api.Address{Location:"", Sublocation:"", LocationType:0x0, Port:0x9b93, Type:0x2, LastSuccessfulPing:0, LastSuccessfulSync:0, Protocol:api.Protocol{VersionMajor:0x1, VersionMinor:0x0, Subprotocols:[]api.Subprotocol{api.Subprotocol{Name:“c0”, VersionMajor:0x1, VersionMinor:0x0, SupportedEntities:[]string{“board”, “thread”, “post”, “vote”, “key”, “truststate”}}}}, Client:api.Client{VersionMajor:0x2, VersionMinor:0x0, VersionPatch:0x0, ClientName:“Aether”}, EntityVersion:0, RealmId:"", Verified:false}, Entity:"", Endpoint:"", Filters:[]api.Filter(nil), Timestamp:1610186188, StartsFrom:0, EndsAt:0, Pagination:api.Pagination{Pages:0x0, CurrentPage:0x0}, Caching:api.Caching{Pregenerated:false, CurrentCacheUrl:"", EntityCounts:[]api.EntityCount(nil)}, Results:[]api.ResultCache(nil), ResponseBody:api.Answer{Boards:[]api.Board(nil), Threads:[]api.Thread(nil), Posts:[]api.Post(nil), Votes:[]api.Vote(nil), Keys:[]api.Key(nil), Truststates:[]api.Truststate(nil), Addresses:[]api.Address(nil), BoardIndexes:[]api.BoardIndex(nil), ThreadIndexes:[]api.ThreadIndex(nil), PostIndexes:[]api.PostIndex(nil), VoteIndexes:[]api.VoteIndex(nil), KeyIndexes:[]api.KeyIndex(nil), TruststateIndexes:[]api.TruststateIndex(nil), AddressIndexes:[]api.AddressIndex(nil), BoardManifests:[]api.PageManifest(nil), ThreadManifests:[]api.PageManifest(nil), PostManifests:[]api.PageManifest(nil), VoteManifests:[]api.PageManifest(nil), KeyManifests:[]api.PageManifest(nil), TruststateManifests:[]api.PageManifest(nil), AddressManifests:[]api.PageManifest(nil)}}

*** The question is: What is the way to have Official Aether to report what of the request is “invalid” ?

Thank you

#2

On a side note, I am not the only one to send “bad” requests

2021/01/09 17:43:17 NodePOST POST request parsing failed. Error: &errors.errorString{s:“The request is syntactically valid JSON, but it does not include certain vital information. Remote: 213.152.186.163:42518”}
, Request Header: http.Header{“Accept-Encoding”:[]string{“gzip”}, “Content-Length”:[]string{“1051”}, “Content-Type”:[]string{“application/json”}, “User-Agent”:[]string{“Go-http-client/1.1”}}
, Request Body: api.ApiResponse{NodeId:"", NodePublicKey:“199debc95630e49dde684601f4fe49705c3ac440ca6e66559fc52e36cb783eb6”, Signature:“4f018e92b7d5385106b471212679e343b29f755494a4948ae479fa3f94b73c8dd0310cc935fc32152fbb493499b64a60d8c6f751cfaf49114e10b5315c65e000”, ProofOfWork:“MIM1:15::::iZOvPFLVaXbrvlbM:16438:4cdbaef532bb063d22b6f0dfc4659d2d5547c0316c13192f2dc7f56ec468875a237eab468d10b83d5f375e0424e693a7000c6b8dcc882e843ea864e05d39b904”, Nonce:“a22eb9708239fcce54a61984f90d0836e4ebe7f8f052449966ae857fa645ba73”, EntityVersion:1, Address:api.Address{Location:"", Sublocation:"", LocationType:0x4, Port:0xa73d, Type:0x2, LastSuccessfulPing:0, LastSuccessfulSync:0, Protocol:api.Protocol{VersionMajor:0x1, VersionMinor:0x0, Subprotocols:[]api.Subprotocol{api.Subprotocol{Name:“c0”, VersionMajor:0x1, VersionMinor:0x0, SupportedEntities:[]string{“board”, “thread”, “post”, “vote”, “key”, “truststate”}}}}, Client:api.Client{VersionMajor:0x2, VersionMinor:0x0, VersionPatch:0x0, ClientName:“Aether”}, EntityVersion:1, RealmId:"", Verified:false}, Entity:"", Endpoint:"", Filters:[]api.Filter(nil), Timestamp:1610210580, StartsFrom:0, EndsAt:0, Pagination:api.Pagination{Pages:0x0, CurrentPage:0x0}, Caching:api.Caching{Pregenerated:false, CurrentCacheUrl:"", EntityCounts:[]api.EntityCount(nil)}, Results:[]api.ResultCache(nil), ResponseBody:api.Answer{Boards:[]api.Board(nil), Threads:[]api.Thread(nil), Posts:[]api.Post(nil), Votes:[]api.Vote(nil), Keys:[]api.Key(nil), Truststates:[]api.Truststate(nil), Addresses:[]api.Address(nil), BoardIndexes:[]api.BoardIndex(nil), ThreadIndexes:[]api.ThreadIndex(nil), PostIndexes:[]api.PostIndex(nil), VoteIndexes:[]api.VoteIndex(nil), KeyIndexes:[]api.KeyIndex(nil), TruststateIndexes:[]api.TruststateIndex(nil), AddressIndexes:[]api.AddressIndex(nil), BoardManifests:[]api.PageManifest(nil), ThreadManifests:[]api.PageManifest(nil), PostManifests:[]api.PageManifest(nil), VoteManifests:[]api.PageManifest(nil), KeyManifests:[]api.PageManifest(nil), TruststateManifests:[]api.PageManifest(nil), AddressManifests:[]api.PageManifest(nil)}}

#3

Looking for possible solution, I found this in server.go

// Rules for the request:
// - http.Request content-type == application/json
// - Node Id always 64 chars long
// - Port has to exist, and > 0
// - Type cannot be 0
// - Protocol subprotocols have to include “c0” (aether subprotocol of mim)
// - Has a valid nonce (by proxy, the timestamp is within our allowed clock skew bracket)
// - PoW is verified.
if r.Header[“Content-Type”][0] == “application/json” &&
req.Address.Port > 0 &&
req.Address.Type != 0 &&
req.VerifyNonce() {

I am puzzled by “Node Id always 64 chars long” since I cannot fnd a usage of that field (no easly, at least). Is the comment “misleading” ?

#4

Ok, found one obstacle, exact string match on the header, now, I am at PoW verification

A key issue is the conversion from Objects to Json, something that is full of pitfalls

I did check that a received Json is converted to Java object correctly and back to Json, byte for byte, including the ordering of fields, so, what may not be correct is the Pow algorithm

source={“node_public_key”:“4139c018f836711e9d9d2790e4f78091e354e061b1a1e2371ac5d8a5200bbd69”,“nonce”:“8d76658bb8f1d223f3111a2619860c7d1b3805bb8e1aba14e899621060fce8b4”,“entity_version”:1,“address”:{“location”:“127.0.0.1”,“location_type”:4,“port”:33555,“type”:2,“protocol”:{“version_major”:1,“version_minor”:0,“subprotocols”:[{“name”:“c0”,“version_major”:1,“version_minor”:0,“supported_entities”:[“board”,“thread”,“post”,“vote”,“key”,“truststate”]}]},“client”:{“version_major”:2,“version_minor”:0,“version_patch”:0,“name”:“Aether”},“entity_version”:1},“entity”:“boards”,“timestamp”:1610276149}
11:56:04 POW calc END pow=MIM1:21::::QOPaPJeiQxcdvVWF:2561962:

Unfortunately, all of pow tests are done against a go structure, not as raw data in, it would be optimal to have a test call directly call
func Create(input string, difficulty int, privKey *ed25519.PrivateKey) (string, error)

so, please, can anybody that has a devel for Aether setup run the above with the given input and report if we have the same result ?

Thank you !!

#5

Please, may I have the correct Json (one that will pass POW test) so I can “adjust” the java part to work with Aether ?

A couple of requests will be ok

Thanks you /b

#6

Please, I am not asking for documentation, just the dump of a correct Json request (the one that is transmitted when doing the POST) that includes the POW, ideally a /v0/c0/boards

Or alternatively, if there is a way for Aether to dump the traffic log in a file, that would be good too.

Thanks /b

#7

Please, any of the Aether developers, can you send me the Json string for a /v0/c0/boards request ?

One that has the POW field filled in, possibly withouth signature and then one with signature of the POW.

Or alternatively, if Aether has the way to dump the original request that comes in to the server, if you could share this special setting. This also should be enough

Or alternatively, if you can make a custom version of aether whith this kind of debug enabled

Thank you !!

#8

Hi there,

I’m currently very busy with dealing with the new influx of users (and this is not my full time job, this is an open source project)— I need to also figure out what you’re asking for, it’s not super clear to me what you’re doing and what guidance you are asking for exactly.

I think one thing you might be able to help me help you is to format your posts properly, as it’s very hard to read and understand right now. You’ve also created successive requests of subtly different questions, and I’m not sure which one to respond to.

Can you tell me:

  • What exactly are you trying to do
  • What is the issue you’re encountering

and be as specific as possible — there are many places where a POST request is made with JSON, that basically describes 1/3 of the whole application. Even POST requests to v0/c0/boards can be many, many types, with different validation rules. If you can tell me what you’re trying to accomplish, what POST request you’re sending, and why you’re trying to do a particular thing (in the context of a sync?), and add the POST request that you could get closest to, I can try tour POST request on my own backend and try to fix it to a working condition, if what you’re trying to do makes sense and it is possible.

Mind that POST requests are expensive to other nodes and they are meant to get ‘live’ data, while most of the transmission happens over cached GET requests saved into the node. In other words, if you’re trying to build your sync logic as just one POST request without checking the GET first and downloading the caches from that remote, you’re making that remote do unnecessary work, and if you do it enough times, those nodes might ban you from overusing their resources. You need to implement the full GET GET GET … … POST sequence, with the POST only reserved for the content that is after the end of the last ever cache, and now.

Also — please don’t post successively, just edit your original post, adding new posts makes it harder for me to understand what’s going on.

#9

Ok, here is what I am trying to do

  • Build a Java version of aether that has a standard web frontend. From the tech point of view it will be pure Java (can run on Win, Mac, Linux) has HSQLDB (again, pure java), GWT as web technology to the client (avoid issues with different browsers)

Beside having the DB and GWT wired, at the moment I can do the following GET and parse the results

  • GET /vo/status
  • GET /v0/node
  • GET /v0/bootstrappers

The next step to actually validate che communication channel is to be able to do nonces, POW and signatures.

As far as I can tell from the source the method that allows me to do it is a POST /va/c0/boards

It should return the list of boards available

An Aether server will accept a “request” if it passes a series of tests, so far I am able to pass all of them up to the pow (proof of work). Pow requires that the calculation is done on a Json version of the “source” that is NOT what it is received from the remote, but it is a version that is parsed by the server, the pow field is stripped, a new Json is created and the pow is checked with the provided pow parameters.

To “get it right” I have to send to Aether a Json that will be reconstructed as golang wish

It can be done, but I need to have at least one version of a correct Json request, so I can “adjust” the Json java “mapper” to behave like go does.

at line 569 of server.go, the original request “b” is parsed into an object “req”, what I need is the request “b”, unformatted, for a v0/c0/boards or if you prefer, for any other POST method

Clearly, if you think that this plan has issues, we could discuss and surely agree on a workable solution.

Summary: I wish to provide an Aether server that is easly portable + uses a standard web browser. A side effect of a “standard web browser” is that multiple users could be connected to the same server.

If I manage to get the basic protocol running I release the code under GNU Affero General Public License and then continue from there, hopefully this is ok with you.

#10

OK, can you send me your incorrect version of the POST request? I’ll take a look and try to replicate it in the weekend and if the issue is in the JSON I can try to get you a corrected version.

#11

This is before POW calculation

{“node_public_key”:“4139c018f836711e9d9d2790e4f78091e354e061b1a1e2371ac5d8a5200bbd69”,“page_signature”:"",“nonce”:“73fdaa52b1873dea708721a745106d1d6a1290b60bee37af9da359da282e9600”,“entity_version”:1,“address”:{“location”:“127.0.0.1”,“sublocation”:"",“location_type”:4,“port”:33555,“type”:2,“protocol”:{“version_major”:1,“version_minor”:0,“subprotocols”:[{“name”:“c0”,“version_major”:1,“version_minor”:0,“supported_entities”:[“board”,“thread”,“post”,“vote”,“key”,“truststate”]}]},“client”:{“version_major”:2,“version_minor”:0,“version_patch”:0,“name”:“Aether”},“entity_version”:1},“entity”:“boards”,“timestamp”:1610559120}

The above produces the following POW
MIM1:21::::DuKYzLoKZWdCngBL:833714:

Yes, I know the public key is missing, but the validation fails before the test for public key

This is the Json that is sent to https://localhost:37107/v0/c0/boards

{“node_public_key”:“4139c018f836711e9d9d2790e4f78091e354e061b1a1e2371ac5d8a5200bbd69”,“page_signature”:"",“proof_of_work”:“MIM1:21::::DuKYzLoKZWdCngBL:833714:”,“nonce”:“73fdaa52b1873dea708721a745106d1d6a1290b60bee37af9da359da282e9600”,“entity_version”:1,“address”:{“location”:“127.0.0.1”,“sublocation”:"",“location_type”:4,“port”:33555,“type”:2,“protocol”:{“version_major”:1,“version_minor”:0,“subprotocols”:[{“name”:“c0”,“version_major”:1,“version_minor”:0,“supported_entities”:[“board”,“thread”,“post”,“vote”,“key”,“truststate”]}]},“client”:{“version_major”:2,“version_minor”:0,“version_patch”:0,“name”:“Aether”},“entity_version”:1},“entity”:“boards”,“timestamp”:1610559120}

If you give me a sample of a “correct” request I can work out what I need to do to adjust it to get it right.

Thank you for your help