In the world of the permissioned DLT business consortia, one of the most important challenges is related to effectively governing and growing the network in order to adapt quickly to the always-changing business realities. Blockchain-based business networks have multiple dimensions and aspects to consider at large scales.

In my previous post “Blockchain DLT Considerations for Supply Chain“ I have laid out some important challenges and considerations to make when designing and planning for a blockchain DLT based network. Even though the concepts of those networks are similar across industries, one of the most challenging applications is in the supply chain context due to its complexity. One important notion of such complexity is the scale of the supply chains that drive large international businesses. For example, a business may deal with thousands of suppliers and partners of different sizes that make a private and permissioned blockchain consortium a very complex network of participants and nodes. Therefore a very important question to answer is how to scale a blockchain DLT based network without falling into an operational nightmare and completely losing the benefits and business viability of forming and participating in such a network. The answer to that question mainly depends on the chosen technology, its ecosystem, and the network governance model design. As Hyperledger Fabric is one of the most matured permissioned/private blockchain technologies I will use it as an example for the rest of the article.

I will build on top of the consortium described in my previous article “Blockchain DLT Considerations for Supply Chain“ including farmers’ co-op, regulator, processor, and retailer as consortium participants (see “Supply chain fruit tracing consortium“). In my example, they run “smart contract” based applications to track fruits from farmers to retailers’ shelves. That is a simple example that could be extended to cover additional functionalities (i.e. from purchase to delivery, invoice, and payment) or applied to different areas like healthcare, insurance, etc.

Realistically speaking large retailers may have huge supply chains with thousands of framers’ co-ops and processors. Furthermore, many cargo and freight companies may be involved to move the fruits from the country of origin to the final destinations that may be located in different countries or provinces. To put it in perspective we are talking about a large number of participants on the network that need to endorse and process transactions on their own nodes operating their own infrastructures. The essential question here is how to design and build such a network.

In the following sections, I will go into a specific implementation of the supply chain consortium using Hyperledger Fabric. Hence a good understanding of HLF terminology is needed.

To learn more about Hyperledger Fabric (HLF) I strongly recommend you can go through the community-maintained documentation.

I will cover below the most important aspects of an HLF based blockchain network implementation for the supply chain consortium described above.

The Network Membership Model

Let’s start with the HLF membership service provider (MSP) modeling. We can simply map each of the 4 organizations (processor, retailer, farmer, regulator) participating on the network to independent HLF MSP each governed by a dedicated set of Certificate Authorities servers (where each organization manages and runs its own chain of trust). The HLF ordering service can be modeled as a set of 4 MSPs, where each ordering service MSP is managed independently by each of the 4 organizations. This means that each of the 4 participants operates 2 MSPs. Even though the ordering service (RAFT based) can be operated by a single party it is still a good idea in many cases to distribute the ordering service between the participants to avoid a central point of control/failure. It is important to mention that a governance process should be set in place and agreed on so that the network can be updated easily when changes are needed (i.e. adding a new participant).

Note: More information on HLF MSP can be found here: Membership Service Provider (MSP) — hyperledger-fabricdocs master documentation

The Channel

The channel in HLF essentially defines a ledger. The HLF channels provide great flexibility allowing the participants to transact on multiple ledgers on multiple networks without multiplying their nodes (the HLF nodes can operate on multiple channels at the same time). This way the data can be segregated and siloed between a particular set of participants without exposing anything outside. The best thing is that cross-channel transactions can be implemented and interoperability can be achieved between any HLF networks.

In our fruit tracing case, we can define and use a single channel (named “tracking“). That is the simplest approach and works just fine for a small set of participants. The channel policy can be set to “majority” so that a change on the channel participants’ list requires majority signatures to commit. In some cases, the transactional data has to stay private between participants using a single channel and still be verifiable by the other parties. To address such requirements, HLF uses a concept called private data collection (PDC). The participants on the same channel may use PDC to record transactions without exposing any concrete data from the transaction itself. It is important to mention that the PDC approach is sometimes confused with the multi-channel concept that in fact is quite different. You may go through the HLF documentation to better understand the difference between PDC and multi-channel approach: Private data — hyperledger-fabric main documentation.

I would recommend considering the business requirements when planning for channels and PDCs definitions. It is rather important to define what data will be shared between participants and what transactions will drive that data. The transactions that contain information visible across all participants on the channel may be endorsed by a pre-selected set of those participants based on the particular business requirements. Technically speaking the HLF smart contracts that track the fruits may require a different set of endorsers depending on their nature. For example, any tracking updates during shipping from the processor factory to the stores may need to be endorsed by the processor and the retailer, where the regulator and farmers may just be committers.

Note: more about PDC can be found here: Private data — hyperledger-fabricdocs master documentation

In the drawing above the participants are connected on the “tracking” channel running their nodes on their own infrastructure based on docker and k8s. The nodes communicate securely using TLS over the internet, the HLF peers and orderers connect over the HLF protocol (grpcs based). Furthermore, anyone trying to connect to any of the nodes without a valid TLS certificate issued by one of the participants will fail to connect at the TLS protocol level (assuming mutual TLS is enabled).

The Network topology

The network topology for the 4 participants above is straightforward. Each of the participants may run their own nodes on their own cloud provider in the following setup:

• 2 peers for high availability, possible distributed between 2 different data centers
• 1 orderer node where one of the participants may run 2 ordering nodes so the total of the orderers can be 5. This way the ordering service will tolerate 2 failed orderers out of 5 to keep the ordering RAFT instance operational. The orderers preferably should be distributed across different data centers.
• If Hyperledger Certificate Authority (CA) is used to manage the certificates then a root CA may be used for the MSP identities and a root CA for the MSP TLS certificates. As those nodes are not part of the transaction processing runtime their location and distribution are not essential for the network performance.
• The choice of the infrastructure and tools to operate the nodes is up to each participant. They have to take care to expose their peers usually through their own DNS hosts and dedicated ports.

Each of the participants may operate additional nodes (i.e. committing nodes that do not participate necessarily in the transactions endorsement) dedicated to queries or for the purpose of having additional peers to achieve better network availability with zero downtime. A participant may operate a single peer, however, such an approach may translate in downtime of the transaction processing when the peer goes offline for any reason (assuming the peer is an endorser). I would recommend always having multiple distributed nodes to achieve zero downtime (especially for mission-critical transactions). Furthermore, a proper backup and recovery strategy should be implemented to address downtimes.

A single participant infrastructure may be quite heterogeneous and spread technically over different data centers in different regions. Please note that the network latency may be a concern when deploying a network of nodes that are geographically distanced. The network speed can be improved and tuned to have little impact on the transaction throughput. For example, if an endorsement from multiple participants is needed then that operation can be performed on peers that are closest to the HLF client performing the transaction (the nodes’ speed and latency are considered on any blockchains, including public networks).

The above setup is relatively straightforward to design and implement. However, the network has to scale out in terms of adding more participants who bring in their own set of nodes. Here are a few important questions to address :

• How to define the HLF MSPs?
• How to scale the ordering service?
• How to scale the channel?
• How many new channels to create?
• How to design the topology of those multiple channels?

Let’s assume the above network has to grow with additional participants:

• Add more than 100 new farmers’ co-ops to the network as the retailer may work with different farmers to supply any fruits or veggies.
• Add more than 10 new processors to the network as the retailer may work with processing factories across different geographical locations
• Add carriers as those companies will be responsible to handle the shipping
• Add customs and ports as they would clear the documentation when importing without delays or risk of fraud
• Many more participants may be added to the network in the future (i.e. financial institutions, international trade organizations, etc. ).

The topology of a large supply chain is complex and that complexity reflects on the design and operation of the DLT network and applications.

I will go through some of the most important design considerations when modeling HLF based DLT networks.

Note: DLT networks are usually decentralized by nature so that there is no single point of control to tamper with the data and manipulate the transactions for benefit of one or few participants.

Membership Service Providers (MSPs)

The definition of the MSP usually depends on the participating business entities and their business structure. Often the MSP entities mirror the business entities. This means that the participants have to define their own MSPs and manage their own Public Key Infrastructure (PKI). That might be a problem though for small participants as not all of the businesses may have IT departments and knowledge to technically handle the needed hardware and software. For example, a group of farmers may decide to use a single MSP and a service provider to manage their MSP and PKI.

The Farmers (small participants)

Technically speaking each of the farmers may have their own specific user certificate so that when they trigger a transaction, the system may record the user context and the data ownership. Such an approach makes sense as many small farmers are usually already part of some form of cooperative entity. The cost to manage and operate their MSP and nodes can be evenly distributed so that each farmer is not financially constrained to participate in the network. The service provider (trusted by the farmers) who manages the infrastructure and nodes is responsible for the security and handling of the administration part of the network governance.

The large businesses (medium to large participants)

The rest of the participants may run and operate their own MSP (MSPs) and nodes. The detailed mapping of HLF MSP(s) to a business entity is defined by the business entity and may vary based on the networks the business entity is part of.

In our supply chain scenario, the 100 farmers may be already part of a few different co-ops. Hence each of those co-ops can be represented as a single MSP operated by a set of trusted service providers. The rest of the participants may have their own MSP and operate them independently. This means that each of the processors, retailers, regulators may have their own IT department to manage their MSPs (or use it as a service offered by a service provider).

The different nodes on the network are managed by the participating parties on likely different infrastructures controlled by independent entities. This is an important aspect as any centralization of the infrastructure and service providers may render the network meaningless. In such a case, a central database solution is very likely the better path to go. For example, running all the network nodes and MSPs on a single k8s cluster is a meaningless approach as well as forcing the network implementation of each MSP to be based on a predefined cloud provider or predefined technology (like a specific virtualization/containerization platform).

Once defined the MSPs and nodes can be re-used on different networks if a party decides to join a new network or leave an old one. The MSPs are usually long-term lived entities that may change in the future.

The next important step is to define the topology of the HLF channels. This is one of the most essential tasks of network design work. It is definitely not an easy one as the channel operation (governance) cost is an important aspect of the network viability.

More details on network governance, channels, and nodes topology are coming as part two of the blog so stay tuned.