Include metadata with solc compiler outputs #3606

swaldman · 2017-01-25T23:27:16Z

Recent versions of solc now append a swarm hash of a metadata file to generated contract code, in order eventually to permit a kind of "reflection" on deployed contracts, as users become able look-up a file that includes ABI, reference to source and compiler-version to verify the code deployed, etc.

This file is quite detailed, including information not available in any of the other compiler-generated artifact. Clients of geth would not in general be able to reconstruct a correctly hashing metadata file from other sources.

This (trivial) PR has geth execute solc with command-line arguments that provoke metadata generation along with the artifacts already expected.

A serious downside of this PR is that it makes no attempt to remain compatible with older versions of solc. Attempts to compile via geth using older versions of solc will fail.

Swarm hashes have been appended to generated code since Solidity version 0.4.7.

GitCop · 2017-01-25T23:27:17Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: 240b6b4
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

mention-bot · 2017-01-25T23:27:18Z

@swaldman, thanks for your PR! By analyzing the history of the files in this pull request, we identified @zelig, @karalabe and @fjl to be potential reviewers.

fjl · 2017-01-26T13:10:15Z

common/compiler/solidity.go

@@ -146,6 +147,10 @@ func runsolc(cmd *exec.Cmd, source string) (map[string]*Contract, error) {
 		if err := json.Unmarshal([]byte(info.Devdoc), &devdoc); err != nil {
 			return nil, fmt.Errorf("solc: error reading dev doc: %v", err)
 		}
+		var metadata interface{}
+		if err := json.Unmarshal([]byte(info.Metadata), &metadata); err != nil {


It would be backwards-compatible if you put if info.Metadata != "" { ... } around the json.Unmarshal call. Please add this.

GitCop · 2017-01-27T08:47:22Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: 240b6b4
Commits must be prefixed with the package(s) they modify
Commit: 9b91316
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

swaldman · 2017-01-27T09:15:46Z

@fjl Hi. I've made the change you requested. Unfortunately, it's not enough to restore compatibility with older compiler versions. The issue is that older versions of solc reject --combined-json bin,abi,userdoc,devdoc,metadata

$ curl -X POST --data '{"jsonrpc":"2.0","method":"eth_compileSolidity","params":["contract test { function multiply(uint a) returns(uint d) {   return a * 7;   } }"],"id":1}' http://localhost:8545/
{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"solc: exit status 1\nInvalid option to --combined-json: metadata\n"}}

To fix this, I think we'd have to make sure to run solc --version once, and then condition the contents of the variable solcParams on finding a version of at least 0.4.7. (We'd want to take care that the initialization and checking of solcParams are performed within a mutex.)

I'd be glad to give that a shot, if you think it a good approach.

(i'm sorry not to "just do it™", but this is the first time I've pretended to program in go, so it'll take me a bit more time than i have just now to attempt it with care.)

GitCop · 2017-01-28T21:23:38Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: 240b6b4
Commits must be prefixed with the package(s) they modify
Commit: 9b91316
Commits must be prefixed with the package(s) they modify
Commit: 87a22c4
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

swaldman · 2017-01-28T22:11:17Z

@fjl OK. So we now check the solidity version prior to the first compilation, and cache the compiler-appropriate solidity command-line params. We initialize/access the cached params under mutex, to ensure there are no race conditions between initialization and concurrent access. The code now behaves correctly under any 0.4.x compiler.

Here is a jsonrpc compilation under 0.4.6, which does not support metadata. A metadata key is emitted, bound to a JSON null. This seems reasonable (although perhaps it would be better to emit an empty string or object rather than null?)

$ curl -X POST --data '{"jsonrpc":"2.0","method":"eth_compileSolidity","params":["contract test { function multiply(uint a) returns(uint d) {   return a * 7;   } }"],"id":1}' http://localhost:8545/
{"jsonrpc":"2.0","id":1,"result":{"test":{"code":"0x6060604052346000575b60458060156000396000f3606060405260e060020a6000350463c6888fa18114601c575b6000565b346000576029600435603b565b60408051918252519081900360200190f35b600781025b91905056","info":{"source":"contract test { function multiply(uint a) returns(uint d) {   return a * 7;   } }","language":"Solidity","languageVersion":"0.4.6","compilerVersion":"0.4.6","compilerOptions":"--combined-json bin,abi,userdoc,devdoc --add-std --optimize","abiDefinition":[{"constant":false,"inputs":[{"name":"a","type":"uint256"}],"name":"multiply","outputs":[{"name":"d","type":"uint256"}],"payable":false,"type":"function"}],"userDoc":{"methods":{}},"developerDoc":{"methods":{}},"metadata":null}}}}

Here is the same under metadata-supporting 0.4.7.

$ curl -X POST --data '{"jsonrpc":"2.0","method":"eth_compileSolidity","params":["contract test { function multiply(uint a) returns(uint d) {   return a * 7;   } }"],"id":1}' http://localhost:8545/
{"jsonrpc":"2.0","id":1,"result":{"test":{"code":"0x6060604052346000575b6077806100176000396000f300606060405263ffffffff60e060020a600035041663c6888fa181146022575b6000565b34600057602f6004356041565b60408051918252519081900360200190f35b600781025b9190505600a165627a7a7230582031c99a93d371621e9ac0eb5ea2071e675994d15d920965501a8ee1c05ab9199f0029","info":{"source":"contract test { function multiply(uint a) returns(uint d) {   return a * 7;   } }","language":"Solidity","languageVersion":"0.4.7","compilerVersion":"0.4.7","compilerOptions":"--combined-json bin,abi,userdoc,devdoc,metadata --add-std --optimize","abiDefinition":[{"constant":false,"inputs":[{"name":"a","type":"uint256"}],"name":"multiply","outputs":[{"name":"d","type":"uint256"}],"payable":false,"type":"function"}],"userDoc":{"methods":{}},"developerDoc":{"methods":{}},"metadata":{"compiler":{"version":"0.4.7+commit.822622cf.Darwin.appleclang"},"language":"Solidity","output":{"abi":[{"constant":false,"inputs":[{"name":"a","type":"uint256"}],"name":"multiply","outputs":[{"name":"d","type":"uint256"}],"payable":false,"type":"function"}],"devdoc":{"methods":{}},"userdoc":{"methods":{}}},"settings":{"compilationTarget":{"\u003cstdin\u003e":"test"},"libraries":{},"optimizer":{"enabled":true,"runs":200},"remappings":[]},"sources":{"\u003cstdin\u003e":{"keccak256":"0x39775f1b915fe2b2d0085bc6b914c8bab108add0d9586641658bcf2edc18d4a5","urls":["bzzr://b62a862c5641f94299190681847046abbd1501187df26508dc95c9630d74fae8"]}},"version":1}}}}}

There is an issue with the metadata. It is correct, but it should probably be emitted as a simple JSON string rather than in its full JSON to avoid ambiguities about ordering, spacing, and escaping. To verify the hash from this output, you have to unescape \u003cstdin\u003e to . A JSON string would be more elaborately escaped, but it could be unambiguously decoded and hashed.

Neither this PR nor the current version support 0.3. compilers due to a separate issue. 0.3.x compilers don't accept - as a token indicating compile-from-stdin. Compiling via geth-jsonrpc with solc-0.3.x, under the current master branch or under this PR, emits neither an error, nor a correct compilation result, but an empty JSON object. Under this PR, solidity version 0.3.5-5f97274a

$ curl -X POST --data '{"jsonrpc":"2.0","method":"eth_compileSolidity","params":["contract test { function multiply(uint a) returns(uint d) {   return a * 7;   } }"],"id":1}' http://localhost:8545/
{"jsonrpc":"2.0","id":1,"result":{}}

Here is the same attempt (with solidity version 0.3.5-5f97274a) under the current master branch:

$ curl -X POST --data '{"jsonrpc":"2.0","method":"eth_compileSolidity","params":["contract test { function multiply(uint a) returns(uint d) {   return a * 7;   } }"],"id":1}' http://localhost:8545/
{"jsonrpc":"2.0","id":1,"result":{}}

I'd be glad to address this issue, but it should probably be a separate PR.

Ensures that we can decode to a set of bytes that will be consistent with the swarm hash embedded in the code, without worrying about ambiguities of spacing, ordering, or escaping.

GitCop · 2017-01-29T06:21:35Z

Thank you for your contribution! Your commits seem to not adhere to the repository coding standards

Commit: 240b6b4
Commits must be prefixed with the package(s) they modify
Commit: 9b91316
Commits must be prefixed with the package(s) they modify
Commit: 87a22c4
Commits must be prefixed with the package(s) they modify
Commit: c10f65a
Commits must be prefixed with the package(s) they modify

Please check the contribution guidelines for more details.

This message was auto-generated by https://gitcop.com

swaldman · 2017-01-29T06:38:04Z

@fjl I've modified the PR to encode the full metadata file as a JSON string, that when parsed according to JSON rules yields the precise metadata file consistent with the embedded swarm hash. (Before the metadata was emitted as a JSON object, creating potential ambiguities spacing, ordering, and unescaping.) Under older compilers (>=0.4.0) an empty string is now emitted.

Under solidity 0.4.8:

$ curl -X POST --data '{"jsonrpc":"2.0","method":"eth_compileSolidity","params":["contract test { function multiply(uint a) returns(uint d) {   return a * 7;   } }"],"id":1}' http://localhost:8545/
{"jsonrpc":"2.0","id":1,"result":{"test":{"code":"0x6060604052346000575b6077806100176000396000f300606060405263ffffffff60e060020a600035041663c6888fa181146022575b6000565b34600057602f6004356041565b60408051918252519081900360200190f35b600781025b9190505600a165627a7a72305820531afd0902b8ec8b3456a3413b3bc302ea2e91deb1beef2f60d571a6e68964b20029","info":{"source":"contract test { function multiply(uint a) returns(uint d) {   return a * 7;   } }","language":"Solidity","languageVersion":"0.4.8","compilerVersion":"0.4.8","compilerOptions":"--combined-json bin,abi,userdoc,devdoc,metadata --add-std --optimize","abiDefinition":[{"constant":false,"inputs":[{"name":"a","type":"uint256"}],"name":"multiply","outputs":[{"name":"d","type":"uint256"}],"payable":false,"type":"function"}],"userDoc":{"methods":{}},"developerDoc":{"methods":{}},"metadata":"{\"compiler\":{\"version\":\"0.4.8+commit.60cc1668.Darwin.appleclang\"},\"language\":\"Solidity\",\"output\":{\"abi\":[{\"constant\":false,\"inputs\":[{\"name\":\"a\",\"type\":\"uint256\"}],\"name\":\"multiply\",\"outputs\":[{\"name\":\"d\",\"type\":\"uint256\"}],\"payable\":false,\"type\":\"function\"}],\"devdoc\":{\"methods\":{}},\"userdoc\":{\"methods\":{}}},\"settings\":{\"compilationTarget\":{\"\u003cstdin\u003e\":\"test\"},\"libraries\":{},\"optimizer\":{\"enabled\":true,\"runs\":200},\"remappings\":[]},\"sources\":{\"\u003cstdin\u003e\":{\"keccak256\":\"0x39775f1b915fe2b2d0085bc6b914c8bab108add0d9586641658bcf2edc18d4a5\",\"urls\":[\"bzzr://b62a862c5641f94299190681847046abbd1501187df26508dc95c9630d74fae8\"]}},\"version\":1}"}}}}

Under solidity 0.4.6:

$ curl -X POST --data '{"jsonrpc":"2.0","method":"eth_compileSolidity","params":["contract test { function multiply(uint a) returns(uint d) {   return a * 7;   } }"],"id":1}' http://localhost:8545/
{"jsonrpc":"2.0","id":1,"result":{"test":{"code":"0x6060604052346000575b60458060156000396000f3606060405260e060020a6000350463c6888fa18114601c575b6000565b346000576029600435603b565b60408051918252519081900360200190f35b600781025b91905056","info":{"source":"contract test { function multiply(uint a) returns(uint d) {   return a * 7;   } }","language":"Solidity","languageVersion":"0.4.6","compilerVersion":"0.4.6","compilerOptions":"--combined-json bin,abi,userdoc,devdoc --add-std --optimize","abiDefinition":[{"constant":false,"inputs":[{"name":"a","type":"uint256"}],"name":"multiply","outputs":[{"name":"d","type":"uint256"}],"payable":false,"type":"function"}],"userDoc":{"methods":{}},"developerDoc":{"methods":{}},"metadata":""}}}}

I think this is probably the best way to go. A case could be made for emitting the JSON object rather than an escaped string: An object is more natural and manipulable, the rules to canonicalize to the hashed bytes are well specified so clients can undo any ambiguity. Alternatively, a case could be made for emitting the metadata as hex-encoded bytes, because that's the least ambiguous way of specifying the hash-consistent bytestring. Escaped-JSON-within-a-JSON-string is arguably a nice compromise -- the emitted metadata remains human readable, but it is easily and unambiguously convertable to hash-consistent bytes by any JSON parser.

I'll leave this here until I hear from you. go can be fun. who knew? Thanks for the opportunity to play.

fjl · 2017-02-01T01:08:34Z

Sorry, didn't find the time to look at this yet.

swaldman · 2017-03-06T22:36:38Z

@fjl i'm sorry to pester, no terrible rush, but a gentle ping.

whether this way or some other way, working on some dev tooling that uses go-ethereum as its service provider, it'd be good to know how i can get solc generated metadata. ideally this would be consistent across clients, but parity's json-rpc documentation doesn't seem even to include eth_compileSolidity, and I can't find documentation at all for other clients. for now, i guess it's pretty much within the discretion of go-ethereum developers. which means you!

i apologize again for the hassle. you guys are a cut above the rest in actually implementing services for dev-tools, which is a real help.

fjl · 2017-03-16T01:59:51Z

Bad news: eth_compileSolidity has been removed (#3740) due do overwhelming support for ethereum/EIPs#209 by core developers. We'll bring it back under a different name if needed.

If your dev tooling is written in Go, you can still use the common/compiler package though ;)

fjl · 2017-03-16T11:20:37Z

Superseded by #3786

Include metadata with solc compiler outputs

240b6b4

fjl requested changes Jan 26, 2017

View reviewed changes

Check for presence of metadata before trying to deserialize it

9b91316

Request solidity metadata only for compiler versions 0.4.7 and higher

87a22c4

Encode metadata as JSON String, rather than as JSON object

c10f65a

Ensures that we can decode to a set of bytes that will be consistent with the swarm hash embedded in the code, without worrying about ambiguities of spacing, ordering, or escaping.

fjl mentioned this pull request Mar 16, 2017

common/compiler: add metadata output for solc > 0.4.6 #3786

Merged

fjl closed this Mar 16, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Include metadata with solc compiler outputs #3606

Include metadata with solc compiler outputs #3606

swaldman commented Jan 25, 2017

GitCop commented Jan 25, 2017

mention-bot commented Jan 25, 2017

fjl Jan 26, 2017

GitCop commented Jan 27, 2017

swaldman commented Jan 27, 2017 •

edited

Loading

GitCop commented Jan 28, 2017

swaldman commented Jan 28, 2017

GitCop commented Jan 29, 2017

swaldman commented Jan 29, 2017

fjl commented Feb 1, 2017

swaldman commented Mar 6, 2017

fjl commented Mar 16, 2017

fjl commented Mar 16, 2017

Include metadata with solc compiler outputs #3606

Include metadata with solc compiler outputs #3606

Conversation

swaldman commented Jan 25, 2017

GitCop commented Jan 25, 2017

mention-bot commented Jan 25, 2017

fjl Jan 26, 2017

Choose a reason for hiding this comment

GitCop commented Jan 27, 2017

swaldman commented Jan 27, 2017 • edited Loading

GitCop commented Jan 28, 2017

swaldman commented Jan 28, 2017

GitCop commented Jan 29, 2017

swaldman commented Jan 29, 2017

fjl commented Feb 1, 2017

swaldman commented Mar 6, 2017

fjl commented Mar 16, 2017

fjl commented Mar 16, 2017

swaldman commented Jan 27, 2017 •

edited

Loading