Since May 2017, the Chinese telecommunications hardware giant Huawei has been developing a set of open-source software solutions collectively known as Project Caliper which, it hopes, will be able to evaluate the performance of different blockchain platforms based on a variety of metrics.
According to a Google Doc ostensibly written by Huawei engineers Haojun Zhou and Victor Hu, the performance indicators that the system can currently measure are "Success Rate;" transactions per second; "Transaction confirmation latency;" and "Resource consumption (CPU, Memory, Network IO,...)." Hu told ETHNews that although the Google document could be edited publicly, the information in it pertaining to the project was accurate at the time that ETHNews accessed it.
The project's goal is to create a "commonly accepted blockchain benchmark framework" that actors in the blockchain space can agree is a fair and accurate way to assess the relative strengths of different blockchain environments. Zhao and Hu, who note that no such framework currently exists, report that the team at Huawei has been discussing the project with Hyperledger's Performance and Scalability Working Group (PSWG), in addition to using the group's "definitions, metrics and terminology" as a guide in designing its solution.
Project Caliper is not "intended to be an authoritative performance assessment," nor is it expected to support the publication of results for the sake of comparing different blockchain platforms anytime soon. Instead, its creators hope that it will help members of the blockchain community test blockchain environments themselves, so that they can be better informed as they choose which environments would best suit their projects.
Huawei will consider the project a success if it "attracts many users within or out of the community to use it as the benchmark framework." By the same token, if a competing project "shows up and the community is more interested in putting more resource on [sic] it over Caliper, a merge should occur and development will be moved to the new project."
The software currently interoperates with Hyperledger Fabric v1.0.5, Hyperledger Sawtooth Lake v0.8, and Hyperledger Iroha, and its developers hope to investigate and begin trying to integrate with "non-hyperledger DLTs" by December 2018.
The project's "adaptation layer," which Zhao and Hu tout as its "key component," is the software that actually interfaces with other blockchains. It apparently has the capacity to "install" and "invoke" EDCCs, also known as smart contracts, and to "query state from the ledger."
In addition to Huawei and Hyperledger's PSGW, the project's current participants include Hyperchain, Oracle, Bitwise, Soramitsu, IBM, and the Budapest University of Technology and Economics.