Papers

Mot nd echnlogy Oppranity Wi Papes | Difuon Fog—Decentralized Federated Ephemeral Edge Computing Platform:

Market and Technology Opportunity Samuel M. Smith Ph.D.'

Original 2018/10/20 Version 0.1, 2018/11/30 Abstract—Th coming tdal wase of perasvedevces bothpersonal and nemet-of Things (o1 il ot ez a fcton of their potential value without an equally pervasive infrastructure for data integration and analysis at the edge where the devices . Cuetly e iegrtio ad s ppens sl i b ssed ghe oy and gt oo, Wo computing infrastructure in the edge, fog computing. The fog is like the cloud only much more distributed. Because fog computing leverages computing resources much closer to the sources (producers, sensors) and sinks (consumers, actuators) of

ot h h ot for bt eIy, highe prformance and owerco,

The distributed nature of loT means that processing and analysis scales better if aggregated in a botiom up hierarchy from loal 1 gabl The das. icgaion s s fo b vl ok agocaiod 1 1 28 s, I e o v o uch 5 possible the existing cloud based tooling for distributed applications but adapted to the unique characteristics of the fog.

Given the rampant fragmentation, stratification and siloization of existing 1oT device level protocols, frameworks, and platforms, the only viable solution is an Internet Protocol (IP) centric approach that enables reliable, secure, decentralized integration across and through 10T silos and strata. This solution is a decentralized federated ephemeral edge computing ameork, pltorn,and madeiplace ordta negraion and anly. This 5 th Iy Fop. The Dion Fo explots and integrates several emerging technologies to provide a decentralized computing infrastructure al scale for personal data bpltins, et of Thiogs (10T sl e of Trings (Lo, ot o Ty Thngs (5T

ndeTems— g, Fog, Computing, Decentaled, Fderated, Plafrn, Ehaneral, Data Economy

1. INTRODUCTION

il ag tchnology i transorming sconomic an soca activiy in neresinly pervasive ways. One exponenialy o s o s atormaton s what ey b el h dtecconomy: 35 34 15,13, 140 T s ceonomy s St e moneiration and tapte of viloe o h dis beng exacic, ansponed, poceied et and s Y he incasingly povdve i Govicesand serarscmploped s and e prio o a orms of day sty These i nobe nt s il hanes: b, wtchet an o bt v ke for digraonand S of s for exeraiment and sharig. 20wl s amient meen 4, 131 fvices ke st comrolled st spskers and i controllers as well as all the embedded, increasingly ubiquitous, sensors, actuators and other devices that

comprise the intemmet of inge (1) In aiion to Gt enaatd b el devics, here Srowth It seneaed b vl Govices, ich as te Ensmension bl i Sos appcatons, cpaclly bl and web aplctons o monior. sack, nd aph olne ety T ervasve dgiation cvoke e ag of il i of dota Flin all v v, il dot . Uik he e il Tt i h senee of e it s o . the Mt mowisl 1471 hower i digl do a5 o o massutements f o eaf wors. Moreoer, rough collsion,miin, and dllaton, e da i, an b ted o il il el vews of reesant seAMens of h ral ok, (e becom, monedsabl data produts o help desion iking, ranscios, s mprove o il an et h uliy o s Tk pspecive poos e qucsions, b will ot e et vlve s o s snd o 1l vt cptrs ocours s, s et oppr s, v s G o exponetal towt i domnd for nwork o Compng S o S v nd S h dts i and how ity e advanigeof hse apporniie”

T orer o anower hes. qestons 1 hlps founersand he curent e of computing nfrsrctr. Mos pressing of daa appens i semrs. Thcs e sty sarshouesTld ih Computn hsdwinsCongsing of ks of smpi servrs {ogeher i Sorage, remwark comectons, and relable power. A Saf of 1 pfessionls manages and mamains Comptng i The hardwaresore, and T ot comprs cmpying st Th common e ol o g o, emer b omping Faaaca 1 th lo. Bcauss 4o cemers ety osted ot avay om b e s o e an o s ot splcuon, e caghe o il & an pton o s et Tors s v i s of s, vt snd pblic (3], A vt cloud 1 cmpose of compute st damsures owned Comrale b an ity slely for s 0wt . Tspcally e serves i ot cod o olaed e el nan el etvor tht T acas o il e, R 3 piae o iy requres 3 ddicaed I 5af o manags e e v Sl e s e ek pace i “omone e st 16 hov thi ey OOned e A

‘Market and Technology Opportunity White Paper 2 arger entity may have its own dedicated data-centers. A public cloud, on the other hand, is composed of servers in data-centers owned und minsged by an entity that rents the use of those servers 10 ofher entities, Typically the servers in a public cloud all belong to the same network, but each server may have its own firewall that limits access t0 tself. The public cloud host enity provides IT stafl to manage and maintain the hardware as part of the rentl fec. In order 10 be able to manage its rented servers the public cloud host IT staff has login access to the operating systems of those serves. There are also Aybrid clouds that have different compositions of private and public components and conirol over those components[36; 551,

Traditionally, public clouds were used in concert with privatc clouds to add elastcity 10 the private cloud in order 1o scale capacity up and down in response to dynamic variations in load. This reduced the need for expensive overcapacity to handle dynamic peak processing loads. Thus, despite being more expensive per unit of computation, public clouds, on balance, are oflen ‘more cconomical overal by avoiding the expense of unuscd capacity. Increasingly morcover, because of the convenience of not having to maintain dedicated computation infrastructure including the associated IT staff, especially for smaller enttis, public clouds are not just being used 10 augment private clouds but are instead completely replacing private clouds. Furthermore the operational advantages of expensing the rental of infrastructure as a service (IaaS) versus maintaining infrastructure as a capital asset s attractive {0 startups, Onee an entity replaces ts private cloud infrastructure with public ts diflcult (0 ever go back. One factor stopping a wholesale migration to public cloud from private cloud are concerns about security and privacy of the public cloud. In one report 66.5% of companies with private cloud are concerned about public cloud securlty threats o sensitive data and 56.9 percent wiere concerned about third party account compromise] 17]. A elated problem is the competitve threat posed by public cloud providers, such as, Microsoft,that also market software applications. Soon after Microsoft acquired the cloud based source code repository service, Github, over 100,000 plus users switched o competing services citing, as one of man, concerns over the confideniality of their source code in tha

t hands of a potential competitor [36; 72],

Historically, in terms of market share for application workloads, private clouds had a much bigger share relative o public clouds. This has been rapidly declining, one estimate is that in 2016 the ratio of private to public cloud application workloads was at 60.9/39.1%. Whereas by 2018 this raio will have flipped, such that the majority 53.8% of application workloads will be in public clouds [17; 241,

‘What makes the frend of increasing public cloud dominance so concerning is that a handful of Fortune 100 providers control most of public cloud infrastructure and that portion is increasing relative to their competition(73]. In terms of revenue, in 2016 Amazon had a 40 percent share, with Microsoft, Google, and TBM together making up a 23 percent share for a tota of 63%. That total share was an increase of 5 percent from 2015 [67]. It is predicted that in 2018 this dominance will grow such that the combined Amazon, Microsoft, and Gioogle share of the public cloud market will be 76 percent and keep growing t© 80 percent by 2020 [74]. Thus while the public cloud is captaring more and more of the overall compute infrastructure market share, the top four public cloud providers, with Amazon much more than the other three combined, e capfuring an increasingly larger relative share of the public cloud market. This trend towards a technical monopoly of compute infrastructure s the data cconomy becomes an increasingly valuable segment of all economic activity is particularly worrisome

From the broader perspective of answering the question of who will capture the most value from the data cconomy, the five largest companies by market cap are Apple, Amazon, Microsof, Alphabet({Google), and Facebook[118]. In May of of 2018 the combined market cap of these five was over 3.7 trillion USD[136] and is increasing. Apple and Amazon for the frst ime cach surpassed the one frillion mark[118]. Notable is that three of the five are also the three largest public cloud providers and all derive a significant share of their value from the monetization of their large private clouds.

‘The most important insight, however, to understand in terms of recognizing opportunitics is not the landscape of currnt computing infrastructure as controlled by a handul of cloud providers, but to understand what the landscape of future computing infrastructurc could become. As mentioncd above, the data cconomy is increasingly driven by data coming from into

met comnected devices. One estimate gives the number of intenet conmected devices at 11 billion in 2016, tripling to 30 billion by 2020, and then almost tripling again to 80 billion by 2025 [82; 139; 140]. The amount of associated digital data is expected to increase from 4.4 zettabytes’ in 2013, to 180 zettabytes in 2023 [82; 139]. By in large the source of this dat is mot from servers i the cloud but from devices outside the cloud. The portion of the intenet that consiss of devices that are not n data-ceners is called the edge {78]. Thus most sources of data resid in the edge and arc called edge devices. The current situation is that in order to process the data generated in the cdge, the data must be fist sent to remote data-centers, ic, the cloud. This adds cost, latency, rigidity, and complexity [80; 104]. These problems are compounded when the results of that processing are needed back i the cdge. Much of the associated data processing is by nature agergation, intcgration, and analysis, which in many cascs Gan be performed in 2 bottom up manner. Consequently much of the processing does not need to be performed in the cloud. Unfortunately however, there s litle or no current computing infrestructure in the cdge 5o the data is sent o the cloud by defiult for processing. Hercin lcs the opportunity! Tn order to handic the coming exponential growth in the amount of data processing, which some liken to & sunami of data [85], there will nceds be an exponcntial growth in the capacity of compuling infrastructurc. Morcover most of the new computing infrastructure could better reside i the edge not the cloud [4; 80; 113; 133] ‘Because most data wil be created in the ede, compute on that data willlikely be driven to the edge as well[146]. Employing the same clowd metaphor, computing infrastructure located in the edge s ofien called the fog because unlike servers in far-away cloud, fog computing servers arc ncarby the sources (producers, sensors, personal devices) and sinks (consumers, actuators, applications) of data [6; 12; 14; 45; 46; 89; 146].

21 ety = 1 wilion gigsbytes

‘Market and Technology Opportunity White Paper 3

Indeed, returning to our original metaphor, the current data rain is bul a sprinkle, but is s00n t0 become a torrential downpour as tems of of billions of new devices are connected to the internet in the net five years, Collceting, mixing, and distilling value Trom this torrential downpour will require new computing infrastructure thatis best located in the fog no the cloud.

‘The big public cloud providers are providing some support for processing data in the fog. They all offer hybrid publiciprivate fog computing platforms. Amazon has AWS GreenGrass which cnables mstallation of Amazon software on private computc hardware for rumning local instances of AWS Lambda [52). Microsofl has Azure 10T Edge which is more ful featured and allows Tocal Azure instances of Function, analytics, and machine lcarning{10]. Google has Cloud IoT Edge which allows private cdge devices 1o run Google software that aliows some local processing [23]. IBM has a beta of software products for its 10T Edge Analytics nitiaive [38; 49]. These offerings do not yel provide truc public fog computing infrasiructure but could be secn s ‘movement in that dircction. Unil there is a viable competitive threat, there is very litle incentive for the public cloud providers o invest in the hardware needed o provide & public fog. The up front capital cost and lcad time for building large data-centers is daunting. Building a new public fog computing infrastructure, if it Tollows the cloud model, wil be cxpensive and time consuming. This poses the questions: will the future of public fog computing be controlled by the existing public cloud providers? Are there opportunitcs to disrupt the public computation infrastructure market as it moves to the fog? The answer is to the latter question is yes. The next section motivates the core elements of our plan for capturing a meaningful segment of the cmerging fog computing markel

1. Tre FUTURE OF PUBLIC COMPUTING INFRASTRUCTURE

Practically speaking, public fog. computing infrastructure does not yet exist, but is inherently potentially much better suited than public cloud computing for processing the data from an cxponentially growing number of edge devices{1331. This preseats both & huge problem and huge opportunity, that is, how to best build the future public fog computing infrasiructure and capture value from it?

‘When referring (o the public compute infiastructure market, the term centralized (deceniral

ized) has two relevant meanings [141], One meaning is politcal, as in the degree to which the ownership and control the compute elements are spread across ‘multiple entitis. The other meaning is architectural, as in the degree to which the compute elements and functions are spread across multiple computer hosts and physical sitcs. To avoid confusion, this paper uses centralized (decentralized) for ts former political meaning and uses the term distributed (non-distributed) for the laiter architectural meaning. Because compute infrastructure that is more distrbuted tends 10 be less vulneruble to failure and more responsive to dynamic load it tends to have much higher availabilty. As a result much of public compute infrastructure is highly distributed, albeit more complex. One atraction of public cloud is that its providers reduce the time and effort for their customers to build, manage, and operate distributed compute systems. Computing infrastructure under more centralized control, however, ends over tme, 1o be more expensive and less innovative s the providers are able 10 extract more value in the absence of competition. Thus, despite the initial cost savings duc to the scale cffects of large centralized clouds, the ability to control pricing independent of costs eventually dominates. Furthermore, public compute infrastructure under centralized control may expose the customers and owners of the associated data to higher privacy and sccurity risks. Centralized control can be a common mode flure source for the distributed system as a whole thereby making it not just less private and secure but also less available[59; 65; 1161,

‘The goal is o find a way to minimize the problems of centralized control without giving up its sale benefits. The best way to achieve the cost savings at scale without having to build at scale is 10 leverage spare capacity, when available, and commoditized components otherwise. The key enabler is a platform that makes available at zeto or low cost unused or under used capacity. The base commodites necded for a public fog computing infrustructure arc space, energy, and bandwidih 10 house, power, cool, and connect the compute servers. Making the servers work s part of a computing infrastructure also requires software to manage the servers automatically as well as to run a marketplace for the associated products and services. The solution we are building s a public decentralized distributed fog computing infrastructure that leverages sp

are capacity, commodiized components, and leading edge soflware o build out at scale, at an otherwise unachievably high rate and low cost. Our approach takes advantage of an procedented comvergence of technologics and market trends which include distributed consensus, blockehain, serverless computing, disposable servers, automated reasoning, zero trust computing, and tokenomics driven network effects, 1o name a fow. The picces of the approach are outlined below.

2.1. Spare Capacity

The base commodities needed for a public fog computing infrastructure are space, eneray, and bandwidth t0 house, poswer, and connect the compute servers. In the cloud these are provided in specially constructed data-centers with dedicated custom built electrial utility power substations and netswork connections. Building a new data-center may take years and may require large capital outlays nof to mention the cffort to manage geographic and political complications. Fog computing infrastructure needs to e close to where the data is reated, in other words, ocated nearly everywhere. One approach some are faking is to build micro- data-centers [69), although viable, that approach still requires purchasing space, power, and bandwidth for those micro-data- centers. Our approach is to use spare space, energy, and bandwidth that s already available i residential, and commercial buildings. Of these the limiting constraint is bandwidth. Fortunately there i an increasing availability of unused bandwidth especially in residences with broadband connections. Bandwidth is often measured in GB (gigabytes) of data transfer. Total available data transfer can be estimated by multiplying the intemet connection speed in Gibps by a time interval in seconds, such as, seconds per month. Average spare bandwidth in GB is then just the average available bandwidth minus the average used bandwidth. Estimates of average and median intemet connection speed vary widely and often combine mobile and fixed

‘Market and Technology Opportunity White Paper 4 connections, the trend, however, is clear, average connection speed is increasing year over year [21; 42; 42; 68; 91; 91; 148]. One. report estimated that in 2018 47 million (37%) of US houscholds will have at Least a 100 Mbps download connection specd [90]. This seems to be an underestimate as the continually updated Speedtest Global Index showed that for the month of August 2018, that the United States, the number six ranked country, had an average fixed connection speed of 100.07 Mbps download and 33.58 Mops upload [124]. The same indox shows that for the same month,the average global fixed conneetion specds as 47.83 Mbps download and 23.02 Mbps upload [124]. Being conservative, an average connection speed of only 50 Mbps provides a maximum total available data transfer of 16.2 TeraBytes per month = 50 Mbps * 2,592,000 seconds per month. Practically speaking 100% bandwidth uilizaton is ot possible. The average broadband residence in the US used 190 GI3 per month in 2016 [60]. The average per user was 106 GB per month [90]. But even with fractional utilization the available spare capacity is Significant. Incrcasing conneetion spoeds above & certain level docs notsignifcantly ncrease usage 35 about 80% of residential usage is for streaming video and the number of videos downloaded is limited by the number of viewable hours in a day [60; 90]. Consequently, the growth in faster connection speeds is resulting in lower latency, faster downloads, and more spare bandwidth. 5y way of comparison, ypical dta transfer llowances for cloud virtal machine compurs are ypically from one 1o a few T3 per month. This indicates that a S0 Mbps broadband connection should have sufficient spare bandwidth to run one or more fog compute servers. Some broadband providers have data caps on total G transferred with an extra fee to increase the data cap. Nonetheless,the average costfor available bandvidih wil sl below even if o longer at zero cost. 2.2. Disposable Servers Ttis notable that spare compute capacity was not included inthe prior section. Many have stated the assumpion that personal laptops, desktops, and mobil devices have ample dle capacity and therefore could b used for the public compute servers [1; 28; 132] We believe that assumption to be largely falc, primrily, because most users will not put up with the inconvenicnce, Security vulnerabilites, and privacy violations tht result from runing extermal compute jobs on the

ir personal hardsware. These concerns sevarcly limitthe actual obtainable capacity of personal devices for public compute services. This limitation means that merely relying on the spare capacity of personal compute devices is ot a scalsble soluton. A more visble approach is o use dediated compute servers instcad. These need o be incxpensive and casy o deploy by non-lech savy building occupants Although the costof commanity servers i largely a function of volume. Sometimes ancilary markets driv the cost of a given class of server to near ts minimum achievable high volume cost, even a low individual purchase volumes. When that happens the class of compute server i a 5o called st spot for price for performance. When that price point i ow cnough to be incidental tothe value ofthe services provided the compute server hardware i essentially disposable. Tn other words, disposable server's haxdware s so incxpensive that i s incidental o the cost of s provided scrvices. Fortunately there is a family of computer that can be beneficially employed as essentially disposable compute servers. This Tamily is at the sweet spot of compuling in terms of an optimal combination of cost, memory, speed, storage, size, and energy. They arc ARM bused computers that un Linu or some form of Nix like BSD. They also support an open source trustcd execution environment [88]. Several manufacturers make these Raspberry Pi like devices [87; 100; 102]. They have credit card siced boards with simillr layout such that the same enclosures can be used for nearly any mermber of the family. They are very low power ( < 15 Watt) ARM devices and often do not need fans for cooling. One available configuration provides 4 cores at 1.5 GHz, 4 GB of RAM, GPU, and 1 Gbps Ethernet at a quantity one pricing of under $50 USD [102]. When fully outfitted with 128 GB of flash disk space, enclosure, and power supply the price is still under $100 USD. A comparable cloud virtual machine costs almost that much each month. This means that the cost of the hardware may be amortized in as little as one or two months. The. usable lfe spun of the hardvare i at last two 10 three years, thus making s ost incidental o the total ietime volue of he computeservices it may provide. These disposable servers combined with the spare space, energy, and bandwidth from existing broadband capable buildings may provide a near zero cost basis for a public fog computing infrastructure. 2.3. Ephemeral Computing

Ephemeral means short li

ved, temporary. Thus ephemeral computing means computing environment which is short lived. An ephemeral compute environment s set up on demand for given computing task and then torm down once the task is complete [27]. Ephemeral computing provides on-demand availability from computing resources that may be 100 volatl, too diffuse, o to0 expensive to be continuously employed. One particular type of computing environment is a viual server Typicaly, a consumer of compute will rent avirtal server and then configure it with software services. Because the server and its services are long running they are not ephemeral. The customer i responsible for maintaining and pdating the server’s software. Recenty the major cloud providers sarted offeing a new class of service called serverless o fincions as a service (FaaS){115]. This new class of service is a type of ephemeral cloud computing environment For example, Amazon offers AWS Lambda, Google offers Cloud Functions, Microsoft ffers Azure Functions and TBM offers OpenWhisk. With serverless there is o persistent virual server that the consumer must maintain, Tnstead the services are run on demand by the serverless environment [5]. The comput infiastructure provider is responsible for maintaining the underlying server infrastructure. This frces the consumer from the having to manage virual server infrasiructur. The consumer mercly specifics the computing veaquirements to run the services in terms of tiggering events, software libraries, memory, computatio, and newwork. Typically sorverless APIs support running individusl functions that can be composed into ful fledgod services or micro-scrvices [83], The functions are execued based on events thatarc specified as part of the serverless configuraton parameters. The functions are run upon event occurtence. This setup provides dynamically available (on-demand) resources to run the functions as more events oceur. Another approach to cphemeral computing is o use contaners that are created on demand to in response to load [19; 37].

‘Market and Technology Opportunity White Paper s These well known applications of ephemeral computing can better oplimize the cost of computing resources and make them ‘more convenient 10 use.

Because serverless divides the functionality of services into small chunks that may be run as individual functions or at most micro-services, the necded computing resources per serverless instance are much smalle, Consequently the serverless approach s inherently morc compatible with the limitcd computing resources of disposable servers.

2.4. Federated Computing

Federated computing is where cooperating distributed servers together perform computing tasks [36). Federated computing coordinates computing resources from disparate locations to provide a shared computational resource. Historicaly, federation ‘has only been used on large computing tasks such as high performance, grid, or big data computing. The approach of building mare complex services from a set of serverless functions i a new type of federated computing. Typically federated compuing has only been implemented with a centralized coordinator. Federating decentralized fog computing resources can be problematic due to the diffuse control of the associated compuling resources and the extremely sparse nature of any single providers's fog resources. This may require federating fog resources among disparate providers; That is the challenge. Localty concentrated fog computing resources, however, may provide the best price for performance for edge computing; That is the opportunity. Fortunately, with an appropriate decentralized architecture, federation techniques can be applied to combine a group of small disposable servers running a serverless framework to perform a bigger computing task, that is, a full fledged cloud-like service, butin the fog.

2.5. Decentralized Computing

Decentralized computing means that the computing resources are under the control of multiple partes. This requires some mechanism that enables these partis to cooperate and caordinate the use of their resources in a single computing infrastructure As mentioned above, with such a mechanism, a decentrlized fog computing infrastructure would make available. new inexpensive computing resources that would othersise be under-utilized due to thei diffuse and volatile nature. Moreover when equipped with federated ephemeral computing software, multple fog devices could be configured to provide the equivalent of a ‘much more powerful cloud computi

ng server. Thus decentralized federated ephemeral computing may have the highest potential for the optimized utilization of fog computing resources 2.5.1. Distributed Consensus

‘The most promising method for decentralized coordination of resources under the control of multiple parties s a distributed. consensus algorithm. Distributed consensus is the process by which a pool of computing nodes come o agrcement about a sequence of events. This sequence might b transactions in a ledger, or entrics in a database, or the order of operations in a state ‘machine, There are three main classes of distrbuted consensus algorithms,these are: Proof of Work (POW) [98; 112; 131), Proof of Stake (POS) [97; 145), and Byzantine Agreement (BA) [7; 15; 20; 22; 32; 33; 40]. There arc many variants of the three main classes of BFT algorithms. What they all share is a featue called Byzantine Fault Toerance (BFT) [15]. Simply, BFT means that that despite faulty behavior that includcs malicious acts by minority of the members, the pool as & whole vl stll come to

By way of comparison, the basic concept of PoW is thal computing nodes competc 1o Solve a cryplographic puzzle. The winner is the first one to broadcast proof of the solution. The winner becomes the author of the next block of fransactions, Variants use different types of cryptographic puzzles or work. Some variants use actusl or uscful computation work. Oftn the type of work is best suitcd for differen types of computing hardware such as ASIC, FPGA, or GPU. PaW is a simple algorithm but s drawbacks are high latency and low throughput

In contrast, the basic concept of Pos i that some verifiable saking function biases the selcction of the node that s the author of the next biock of transactions. One way 1o think of PoS is as an asymmetric proof of work where a larger stake reduces the amount of work. Thosc with larger sakes win fuster and more ofien. Thus PoS has lower latency and higher throughput than PoW, albcit with more complexity to manage the staking. A popular variant of Pos is Delegated Proof of Stake (DPoS) where participants vole their take (o elect the author(s) of the next block of ransactions [35]. Another el known varianl is Federaled Proof of Stake (FPoS) where participants scloct a subsct of nodes o come to consensus on their specific transactions [32]. Further variants are sharded or hierarchical PoS where the nodes are broken up into sub-groups (shards) that form consensus on & subsct of transactions

and then merge the subscts 142,

Predating both PoW and PoS are the BA algorithms. Tn BA algorithms the number nodes in the poo is fixed. The first popular usable BA algorithm was called Practical Byzantine Fault Tolerance (PBFT) [20]. In PBFT, the nodes in the pool clect a leader, The leader then authors the next block of transactions and submits it to the pool for validation using a threc phase commit. The main advantage of BA i that compared to both PoW and PoS, the latency is much lower and the throughput is very much higher. The disadvantage of BA is complexity and the number of pool members much be kept low. Several variants of PBFT have been developed. Notable is Redundant Byzantine Fault Tolerance (RBFT) which is more resilient [7). Others include Fast Byzantine Consensus (FBC) [77]. Others use some type of dirccted acyclic graph (DAG) to that makes the consensus leaderless and more asynchronous [71; 114]. Finally a hybrid variant of BA and PoS, which we call Randomized Byzanine Agreement, uses a verifiable random function to come to conscnsus [43; 99]. Varianis include biasing the random function by the stake Lo sclcet the author(s) of the next block or ofher similar ideas [8; 50; 103; 135). In the last two years there has becn an cxplosion in the number and type of variants of BFT distrbuted conscnsus algorithms [26; 149] Tt s beyond the scope of this paper to mention all of them or to describe any of these in detail

‘Market and Technology Opportunity White Paper 6

‘The design of a BFT implementation requires making tradeofls between the features of the different algorithon classes. The sulient features that usually onc must rade-oif are sccurity, scalability, and governance. There arc four primary design clements that a BFT distributed consensus implementation must specify and provide, these are:

• The governance policy for selecting pool members

• The distributed consensus algorithm(s)

• The data stracture holding the events managed by the pool.

• The reward mechanism for incentivizing participation of the pool members.

‘The resultant distributcd consensus system s then used 1o provide transactions that support a given application or applications. A systm design task is to composc the best combination of these four clements to best mach the transactions for the application at hand. For example onc very common data structur used in distributed consensus systems is a ledger. Because the pool members are distributed and cach has # copy of the ledger it becomes a distributed ledger. The ledger i ofien implemented as a blockehain to cnable integrity verification. It is notable that the term blockchain is now commonly used to mean more than just a fype of data structure [129]. s more general meaning refers to & host of technologies thal include eryplo-currencies, distributed ledgers, distributed consensus, modern crypto, and decentralized identity to name a few. Solving a business problem using blockchain in iis more gencral sense means selccting and integrating @ speilic mix of the aforementioned technologies. The particular mix is typically targeted to the application that solves the busincss problem.

“The most popular BFT distributed consensus implementations arc PoW distributed ledgers. These includes BitCoin and ‘Ethercum [13; 41]. The altractive feature of PoW i that the pool govemance policy is included as part of the PoW conscnsus determination. This allows for open public participation of an indeterminate mumber of pool members. As prviously mentioned, the main limitation of PoW is high latency and low throughput. PoW by itself is not scalable 1o thousands of transactions per sccond. Consequently much of the recent innovation in bitcoin techmology has been 1o find ways to address the latency and throughput limitations of PoW. One solution is o use a PoS system, which usually has higher throughput and lower latency than PoW but may requirc more carcful manag

ement and governance in determining the stake. As previously mentioned, PoS can be thought of as an asymmetric PoW where the size of the stake determincs how much work must be done. Tn contrast, BA algorithms have the fastest throughput and lowest latency of any of the three algorithm types. The main drawbacks of BA are its complexity and the total number of pool members i limited.

2.5.2. Hybrid Systems

The approach that we believe has the most promise is to use a hybrid system. In a hybrid system, muliple distributed consensus mechanisms and pools given & more optimal combination of securiy, scalability, and goverance. For example, one way (o achicve both high sccurity and high throughput is 1o usc a PoW ledger as a periodic anchor to transactions that happen on another faster non-PoW auiliary distributed consensus pool. This is often called a side-chain [2]. Another approach s to use a different consensus mechanism for pool governance than for transuetion ordering. In the fog computing application, computc job allocation must be completed with low latency and at high throughput, Cansequently, the only viable choice for the auxiliary consensus algorithm s a type of BA algorithm. Because pool membership goverance is not built into BA algorithms, using BA requires implemening a governance policy o algorithm to manage pool membership. Because pool membership changes are relatively infrequent event, however, a slower consensus algorithim such as PoW/PoS can be used to automate pool membership govemance. Periodically anchoring the BA pool cither to a much lirger or to multiple PoW/PoS pools provides increased security while sill enabling very high tansaction rates, We originated a form of this called ChainViail [108]. A distrbution algorithum such us network distance or somme type of distributed hash table (DHT) to assien transactions to dillerent pools makes it scalable. We believe thatin general this s the most viable approach for decentralized fog computing infrastructure.

We have deep experience with BA algorithms. One is Plenum, a modern BA implementation based on the RBFT algorithm hosted by the Hyperl.cdger Indy project (95]. We have also originated a next generation BA algorithm called TOBA (Timeliness Ordered Byzantine Agreement). TOBA s a leaderless asynchronous hybrid algorithm that combines elements of a conventional ‘multi-phase commit with a directed acyclic graph (DAG) to make it both more resilient and more scalable. One advantage of TOBA is

that the core algorithm lends itself to an automated reasoning algorithm that ageregates proof of useful work, delegated proof of stake, and expected reliabilty to sefect pool members (governance).

2.3, Design Issues

A careful examination of the scalability proposals for PoW blockehains such as Ethereum and Bitcoin reveal that scalability is not possible with a single distributed ledger that tracks all transactions as a single asset class. Proposed fixes include some combination of sharding (multiple asset classes with receipts between shards) or side chains [2; 142]. A more scalable approach i to use triple signed receipts on transactions amongst mutually disjoint sets of parties to the transaction [S3; 54; 84; 101 125 136; 143]. The receipts provide non-repudiable proof of the agreement at each stage of a ransaction. The consensus pool acts a trusted third party to sign the receipts that clear transactions within a given asset class or between two or more asset classes. These completed transactions can then anchored to @ slower cryptocurrency based blockchain such as BitCoin and thereum.

A decentralized identity system provides the core mechanism for allowing interaction between different pools and different asset classes. A performant asynchronous peer-to-peer communications protocol that uses decentralized identity makes the approach scalable. Both TOBA and the first implementation of Plenum use the RAET (Reliable Asynchronous Event Transport) protocol [107]. It s based on NaCL (LibSodiam) and CurveCP which are recognized as the state of the art open source ECC

‘Market and Technology Opportunity White Paper 7 crypto libraries for key exchange, signing and encryption. RAET is an inherently scalable UDP/LP based protocol that uses best prctices for sccure relisble communications. We originally developed the open source RAET protocol for highly distributed cloud infrastructure automation, so it is well suited for fog infrastructure automation. One of the primary concerns of fog computing infastructure is the management and control of the computation and data on remote edge devices. Management of these devices requircs a sealable reliable sccure communications protocol. RAET is that protocol.

Building disuributed applications requires interoperable. communication protocols 1o conneet. software componets on disparate servers. While there arc many communication protocols that may be employed 10 connect software components, the most universal approach is t use 1P (Intemet Protocol) based protocols. Indeed, one of the biggest challenges in fog computing for LoT applications is the fragmentation, stratification aad siloization of existing IoT devices due to the rampant use of non-IP based protocols. Historically, the limited computing resources of embedded 1T devices motivated the used of resource constrained protocols, but every year the fronter of applications that are too constrained for IP protocols s shrinking. The CoAP projectis an exumple of lightweight LP implementations that are [rendly for consirained resource networks [25]. Thus nop-IP networks are now generally only desirable as legacy point optimizations, not universal solutions. The only viable universal solution for a public fog computing infrastructure s an Intemet Protocol (IP) centric approach that enables reliable, secure, decentralized integration across and through legacy 10T silos and strata. Consequently Difuon s IP based. Aceess toffrom other data sources and sinks that are not IP based may be provided via gateways. Moreover Difuon's IP centric nature makes it very convenient to leverage a multtude of cloud based software tools, protocols, databases, frameworks, and platforms,

Many blockchain applications involve some form of smart contract, that i, rules that execute in trusted but automated way to manage a given contractual relationship. An important consideration in this contract management is that the rules be Verifiable. We have developed a flow based programming and decision-making framework called loflo that uses a model based approach t

o program synthesis and analyses that allows for easier verification [105; 106]. This was originally developed for autonomous vehicle systems. Toflo is asynchronous and thercby provides framework for building scalable backends for data processing of transactions. loflo is a type of flow-based programming framework with many unique features that make it applicable to highly distributed asynchronous reactive systems [18; 31; 44; 57 63; 64; 70; 92; 127; 128]. Difuon repurposes it to provide a fog centric serverless computing platform. The curent "serverless” frameworks are cloud centric and largely proprietary. Because it i a flow-based programming framework it is uniquely suitable for IOT systems. loflo itself is mature but the repurposing as a dynamically allocated functional service is new

Important to any blockehain application is privacy preservation. Typical fully public blockchains are problematic. The path forward is to use public information as a trust anchor but the vast majoriy of information is exchanged in pairwise transactions that are off-chain. We are innovators i this approach. Decentralized autonomic data (DAD) items based on DIDs (Decentralized Identifiers) enable streaming data in a scalable but secure manner [34; 110; 117]. Using portable W3C Decentralized Identifiers (DIDs) provides the foundation for an open decentralized trust system. This enables a portzble decentralized atribute based identity for every entity and every thing including every item of data. Decentralized Autonomic Data item (DAD) chains build ‘with DIDs provide sclf-contained end-to-end data provenance of cach data flow 2.6. Diffuse Trust Computing

Federation requires that disparate entites cooperute 1o share resources, Shring resources raises the risk that data or assets ‘may be compromised because the of the insbility of onc cniity 10 trust the actions of another. The conventional approach to IP sccuriy is often described as defense-in-depth where cach nested layer forms a sccure perimeter that restricts access to higher levels of trust. These perimeters typically consist of some form of host:port bused firewall. The outermost layer faces the "wild- wild-west” of the open intemet where cvery interaction may be hostile. The next layer may be in a DMZ andor behind fixewall, The following layer might be a trusted servies running on a private network. VPNs provide a way for an cxtormal remotc host to tunnel through the outer layers (o access a truted space. What a per

imeler model assumes is that nctwork locality can b trusted. Tnternet hosts and scrvices running in an inner layer arc allowed to sharc information with cach other because they are in a trusted “safe” space. This conventional approach fals for many reasons. Fundamentally it fuls begause it docs not account for the fact that hostile uscrs exist cverywhere ot just on the open intermct. Any mechanism that allows for lower privileged users o access @ network layer provides an attack vetor for that user 10 elevale their privileges thereby gaining aceess o everything in that layer. Henc, network locality is not sccurity. Morcover, perimeter sccurity s sttic and docs not respond to dynamically evolving exploits.

“The solution is to use diffcse trust perimeter-ess sccurity model. This is an extension of an earlicr approach known as a zero- trust security model. A simplc way of cxplaining the zero-irust sccurity model is the mantra, never frust, ahways verify. The paradigm of Zero-Trust Networking was frst popularized in 2013 by the a NIST report [47]. More recently the principles have received much broader atication [S1; 144]. Calling it zero-frust or trust-les is a misnomer. There is still trst, 1t s just applicd in such a way that security is enhanced. We have extended this model to accommodate computing resources under decentralized control [109; 110; 117, We call this a diffse trust perimeter-less security model because the secutity policy is maintained with decenralized resources that diffuse the trust in such a way that security is even more greatly enhanced than merely 7ero-{rust, A computing infrastructure that is built from the ground up using a diffse rust perimeter-less security model, We call difese-trust ‘computing. We belicve this is the only truly viable approach for a public fog computing inffastructure. To our knowledge, Difuon is the first decentralized public fog computing infrastructure based on diffse trust computing principles and may be the first public computing infrastructure of any kind based on these principles

The assumptions and principles of diffse frust perimeter-less security are: (see [109; 117))

‘Market and Technology Opportunity White Paper 8

• The network is always hostile both internally and extenally; focalit is not trustworthy,

• By defaull, inter-host communication must be end-to-cnd signedencrypied and data must be stored signediencrypted

using best pracices cryplography; Data is signediencrypted at motion and at rest,

• By default, every network interaction or data flow must be authenticated and authorized using best practices cryptography.

• Policies for authentication and authorization must be dynamically modified based on behavior (reputation).

• Policies must be governed by distributed consensus.

• By default, cach data flow including all ransformations must be end-to-cnd provenanced using decentralized identifiers

(DIDs) and hence decentralized autonomic data (DAD) items.

The first stcp to implementing diffusc trust computing principles is the recognition that any [P network is always hostile. This ‘means all servers must be hardened using best practices for ports and services, Using end-to-end encryption and torage prevents exploits from anyone that mercly has access to the nctwork or the data storage device. By authenticating and authorizing cvery network interaction or data flow, security becomes granular. A successful exploit of one interaction does not bleed into any other. Compromising one data flow docs not compromisc any other. Escalation opporlunities arc minimized. Many sceurity exploits are discovered through repeated probes and experiments to find bugs, buffer overflows, or weaknesses in network protocols or software implementations, Dynamic policy modification that uses & reputation Al to first profile and detcct anomalous behavior and then restrict the authorization of that uscr prevents discovery. This adds time as a defensc. Extending the model to use distributed consensus o goven the policy management including authentication and authorization enhances the security over using & centralized policy manager. Distributed consensus diffuses the trust for any policy decision to a group of hosts. In order o defeat the policy, an attacker must exploit some majority of the hosts. This makes exploits exponentially more difficult. Morcover. distributed consensus also allows for the use of decentralized computing resources for hosting the policy munagement pool and decentralized governance of the hosts. The final principle further exionds the decentralized trust model fo allow govemance that leverages a decentralized web

of trust based on the emerging W3C open Decentralized Identifier (DID) standard and Decentralized Autonomic Data (DAD) items bl with DIDs [34; 110; 117, These provide provenanced data flows that arc compatible with big-dala, data-flow, reactive, and flow based programning approaches needed for distributed data analysis tasks [18; 44; 55; 57; 63; 64; 70; 79; 92; 105; 106; 110; 117]. This approach cnables truly decentralized governance models for distributed applications. One can combine the third and sixth caveats from above into a simpler summary caveat a follows:

• By default all data flows are end-to-end provenancedsignedencryped at motion and at rest using DIDs and DADs.

The diffuse trust security model by itself addresses the security concerns of a_decentralized public fog computing infrastructure. The other two main concens are privacy and agency. We call these the SPA (Security, Privacy, Agency) design principles,

• Secuity

• Privacy

• Agency

Sccurity, privacy, and agency are all cxtremely important lo 4 decenalized public computing infrastructure. Many implementations only atiempt (o address one or two of the three. Difuon s designed to provide reasonable levels of all three. As previously mentioned distributed consensus helps improve sccurity in the open.

Privacy is about minimizing the disclosure and leakage of of information to uninvolved third parties that may exploit it harmifully. For example, & log computing trensaction is typically between two diflerent partes, the compute provider and the user of that compute. In order o consummatc a transiction, the partics may need to share information about cach other before proceeding with the transaction. In addition, the transaction itself may involve the exchange of information. This is obviously the case for fog computing transsctions. A modular approach 10 privacy allows the participants (o exchinge on-chain (public distributed consensus pool) as litle as possible and exchange the bulk off-chain (private peer-to-peer). A detailed exposition of privacy concens and methods s beyond the scope of this paper but the core principles may be found in the provided references [108; 111; 122, 123], Although the citations include white-papers from the Sovrin Foundation, the Difuon Fog is not an implementation of Sovrin but is based on similar principles such as privacy by design, minimum disclosure, and pair-wise unique identificrs for cach relationship. Like Sovrin and others in the Decentralized Iden

tity Foundation (DIF), the Difuon Fog does use the open portable DID standard [34; 62].

What is worlhy of further explication is the degree of privacy preserved by the systom. Ono way to classify privacy preservation is as being ither sirong or weak. Sirong privacy preservation is achieved when any third party correlator can not practically extract any eflective correlation between disparate identifiers used by an entity across multiple trunsactions. Theorcically one could contcmplate sirong privacy preservation for caitics that follow a policy of a one time interaction with any other entity and the set of atiributes used in all their interactions s mutually disjoint with respect t the intercting enttics. ‘However it would be difiult for an entity prcticing such behavior to find other entities willing t0 intcract with it. Thus making such a policy impractical. Generally speaking, any entity that regularly interacts online with other entites will eventually be correlatable because many hird party correlators have practically unlimited storsge, compute power, network access and time o make correlations. Consequently, we believe strong privacy preservation s not practcally achievable especially for entitis that

‘Mirke nd Tchvology Opportniy Wit Paper . gain value from regular online interaction. In contrast, weak privacy preservation is achieved when the cost of correlating disparate identificrs is more than the valuc exploitable from that corrclation. This minimizes the cconomic incentive for third party correlators to engage in correlation, Weak privacy preservation, therefore, is an economic problem and is practically achievable for many applications. Nation state correlators may not be inhibited by economic considerations for a given entity but oty b nbbied A e mmbor of et i g enough. Thes ek prvacy presevon b ot bt achcvible on csgt wil respect to nation state correlators and may be achievable individually for correlators with much more limited resources. One of the contributing factors is that both the value to the correlator and the harm to the correlatee tends to decrease the longer the time: e aher s comlnable meraclon hos ocured. Thos wedk rvacy presving sysems san ve sgnicu empors Somponsas

‘Agency deals with the control and governance of the associated data and transactions. In fog computing, centralized governance is problematic, Distributed consensus algorithms provide a practical way to implement decentralized governance. Indeed as the foregoing discussion has shown, using distributed consensus to manage security, privacy, and agency provides a bencca snety 2.7. Notional Architecture

Thi secon provides an overview of basc rchitcture of the proposed fog computing nfasucture, 0 includs brct dcerions Fthevaious Somponens and the e they play. o whieepers wil xplin i more deph e ssocined ool i lgorims. e Foloing dingra Drovide & SapHia repesciation o the sarons P g componens R own bl fo conputing i

8 !’ = = —mQ Wi VAR 7T\ /N L \ / P & \ / - B CORNT. "’ | C AP \ !6 .\ /Eg \ e \ \ £ B, \/ N/ 5 ;ss;s;sv:;g@ S o i —a

O 271, Disvibuted Consensis ool

“The core component is a Byzantine Agreement (BA) distributed consensus pool. The main feature of the BA pool is that it provido rad doteted Sonsena it ook he abIHY 1o uppen. et Saclons. By mae amsactons e e Tnacions htsetle vary ekl and for very small amounts. Tl exble ekl i he compostion ad remmeraion o e seve group o provite sphemers e snvices and fneions b i (Faa) Fomaly hse rmacions il e wackadon s drbtad s of accoun, batanen aained byt I poo. Th provies the bt o poyment. i v th Tedae of accunt e BA pol will i it dasbas. The BA pol provdes aron sarmecs o omiency i s s consen s, This s o othr modTeons h

deer ordatabe oy oot 112 it majority o pool e re I ageement. Commequonly 4 ong 4 oy minorty ofpoo metmhes e Ty on alciow a1 an. i, modeaton. 1o e Tedger nd daibue wil e enonhy Moreovce he proces of enesng oo i iher he Tedger on poo reat i the generaton of o Al gnawes on e comens of shoe raetion. The ramsacions cin e vrihed by redin he comeon of 1 dosboss and dge The dtabose ol vck engagement and on-osndivg of paricipats i he. ptforn i tht DIDs (Deeenlzed o) and smocised Ky o e dtabase i s privide uppor Tor (b smote gt of compoling Geiccs by providn » s e sy gt whieh (h conTraton of ay node o b verHed The datibane will s s cpabiy andpeformance bute o) he patiipaing hows g arbote bascd srou. nemwork locuon nd dsancs Sone st o e e 0 g bl fdeonaroupnes t ommmee ofcompute. T pol il s e bkt matehes avatale ompin e P 1 e demand O o e s alched th pos s e et ehongh vt o the

‘Market and Technology Opportunity White Paper 10 consumer (o pay for the ordered compute services. The escrow is released 10 the compute producers once they provide a proof of completion. The distributed consensus pool minsges the setllement and finalization of orders as they are completed. Al entitics be they partcipants, hosts, orders, o any data item (DAD) are tracked with DIDs or derived DIDs (dDIDs) [34; 117]. This provides a universal mechanism for cstablishing the provemnce of transactions flows in the system. The purpose is to provide credible processing chains in a decentralized control enviromment. This extends conventionl centralized distributcd strcaming data axchitectures such us Kafka, Lambds, and Reactive programming 1o decentralized compute infiastructure in a secure and ‘manageable way [55; 70; 79]. The distribution of credentials for marshaling scrverless frameworks will also be bascd on the extensible properties of DIDs. Using the emerging W3C decentralized identifier (DID) standard as the basis for the distribution of credentials for marshaling cphemeral containers and the provensncing of data flows is @ unique featurc of Difuon. This infrastructurc will support simplificd clicnt key management techmiques for key reproduction, pre-rofation, and recovery. This identity layer enables credible reputations of paricipants on the platform.

‘The BA pool also prescrves privacy by using a combination of group capability atiributes and blinded routes. Group capbility means that all members of the group share the same minimum capabilitics Thus the BA pool can make matches from the order ook with a group of viable compute nodes that have the minimum capability for the transaction without indicating a specific ‘member of the group. The requesting node then selects off-chain the specific member(s) from the group of potential providers it wishes 1o engage with for its compute 1asks. An attacker would have (o simultancously exploit all the members of the group thercby making the exploit exponentially morc cxpensive. A blinded route uses blinded signaturcs to provide verifiable but not publicly traccable network addresses to the potential participunts in a ulure computing trnsuction [L1; 30; 39; 134]. This is a type of zero-knowledge proof [150]. This i similar to how blinded signatures arc used in E-Cash and E-Voting applications [16]. “Thus each participan, source or sink of dta, can verify that the route to cach other provided by the BA pool s vali

d from the order book but the BA pool has no exploitzble knowledge of the routes. As a result, the data may be transferred and the computation performed off-chain in a pair-wise private manner, Once the computation is complete, the exploitable value of the data exchanged is minimized. In addition, using self contained DADI based blockehins enables full end-to-end traceability and verifiability of a computation flow. Each member of a BA pool must be ablc to communicate with every other member. This is shown symbolically in the following diagram.

Fig 2. Single BA dinrbuted consenus ool

To improve performance and minimize latency, multiple BA pools will be cmployed, cach in a given zone grouped by network distance. Network distance may be measured by the round tip time, . latency for packet o traverse the network from one host to another and back sgain. Usually the network distunce is a function of the number of router hops it encounters along the path. Because routers tend 1o be geopraphically distributed, using network disance also tends to result in a geographic distribution of BA pools. This is not exact as local data-centers may have high speed low ltency hops to more remote data- centers. Offcn the latency is a function of the cost of the comnction. Using network distance allows sources and sinks of data o e matched to minimize average latency and or cost or some combination of the two. The number of BA pools will expand as the network grows. The set of BA pools forms a distributed partally overlapping partal mesh of pools where the partisl overlap between pools is with respect 10 that poof's nearest neighbors in network distance. The partial overlap provides redundant aceess by a given participant to muliple BA pools to incroase aveilability and minimize common-mode fuilures. The partial mesh means that some pools can anchor their iransactions with respect to some other pools but not all other pools. This is the chain- ‘mail approach to increasing security without detracting from throughput [108]. In addition, the partial mesh means that some pools may make transactions that infersect participants between two different pools. This i a type of sharding. A hicrarchy of ntersecting pools will allow any set of participants to mutually engage in transactions albeit at increased latency. Fortunately the actual data transfer for compute happens pair-wisc peer-1o-peer of-chain and therefore ts ltency is not affected by the hicrarchy of intersecting pools. T

he demand for cross pool transactions is minimized because the vast majority of fog computing interactions will benefit from nearer network loality. The following diagram shows a set of neighboring BA pools in a partial mesh where each interacts directly with its three nearest neighbors and can also interact with the infersection of a s of paols fo enable indirect mutual interaction of all participants.

‘Market and Technology Opportunity White Paper n Fig. 3. Systom of geographically distibed BA pools 27.2. Relay Network

Augmenting the netswork of BA pools is a relay nefwork of server nodes. One function of the relay nefwork is provide read- only access to copies of the distributed ledgers and databases maintained by the BA pools. The relay netwark thercby reduces the read load on the BA pool members. As previously mentioned the main purpose of a BA pool is to provide a trustworthy verifiable mechanism for maintaining consistent updates to the distributed ledgor and database managed by that pool. These data structures are designed such that a reader of a copy of the associated ledger or database can verify the consistency of the contents via the included signarures. Because the majority of accesses by users to the ledger and database will be read not write, the relay. network improves scalability by both balancing the read traffic across the relay hosts and reducing the read traffic on the BA pool members. Because a relay node's copy of the ledgers and databases is read-only the only exploit a malicious relay node can ‘perform is to delete or ignore updates to the associated contents but not modify those contents. Thus any user merely necds to be able to access one non-faulty relay node to be able to read and verify updates to the ledger and databases. This non-faulty relay node could be under the control of the user or some other trusted entiy. Each user may select a sct of relay nodes to provide higher availability. To restate, write updates require a trustworthy non-faulty majority. Whereas read verifies merely require a trustworthy minority — as litle a5 one.

In addition to providing load balancing readiverify copies of the distributed ledgers and databascs, relay nodes may support other functions. One of these is to provide a software defined data flow routing overlay [109]. This i similar to software defined networking but decentralized [119]. This maps data flow addrosses to IP addresses. Thus a set of compute services or functions can be pre-configured with data flow addresses that only need to be converted to the real TP addresses of the actual hosts at run time. The relay nodes maintain a data flow route to [P routing table. This allows the serverless framework to dynamically allocate compute resources without reprogramming or reconfiguration of compute services and functions.

Another funcion of the relay nefwark is to bootstrap peer-to-peer

communications befween nodes in the nefwork by ‘maintaining a DID-to-IP host:port mapping table. Most TP nodes in residential and commercial building networks do not have public TP addresses but have only private P addresses and reside behind NAT routers. The default configuration of most NAT routers enables dynamic port mapping (forwarding) [96]. Dynamic port mapping opens a retum port on the extemal (pubic) side of the NAT router as a resul of an outgoing packet from the internal (private) side. This allows an internal host that does not have a public IP address to initiate two-way communications with an external host that does have a public IP address. It does not allow the converse. This is a problem for private hosts that want to work as compute servers. Private hosts can query the DID-to- TP address table on relay network hosts that have public P addresses to helps bootstrap peer-to-peer commmunication with another private host. The basic technique is called UDP hole punching [137). RAET is well suited to this approach. Hole punching is less eliable with TCP connections [130]. n the event that hole punching does no work, two nodes may communicate using the relay node as a repear.

‘Another function of the relay network is to provide a content and code delivery network (CDN). Client side web applications require a static Ioad of content. Having this static content cached in nodes that are nearby the web clients reduces starfup delay and minimizes bandwidth demands. Morcover pushing updates to compute nodes code and frameworks can induce spikes in nefwork traffic. This is minimize with cached local copies. Video and audio streaming applications consume large amounts of ‘bandwidh this is also minimized if the content is cached locally.

Another function of a relay node is to facilitate federation of compute services. The relay nodes keep a local copy of the federation groups for compute services. The relay nodes track also track presence, availability, and performance of any of their conneeted or communicating nodes. This allows compute consumers to better dynamically select the specific servers to use from their federation pool.

In general the relay network's purpose is to facilitate and improve performance, horizontal scaling and greater elasticity of the computing infrastructure. The number of relay nodes is not fixed and may exceed the mumber of BA pool nodes by an order of ‘magnitude. The following notional diagram shows a BA pool with auxiliary r

elay network nodes.

L R B::d] ‘A‘»é«'gh - — 177 N .\" 0 ssne i e s i (O Thr s e P

Ll\ i ‘ll‘ P o —¢ 98 “‘-.‘.-'.‘ SRPANNEC A\\~\' LTS P/ Al avall A fifl..‘,,q,“',‘ii i i BN AN c;” \2§J The final participant role are loT devices such as sensors and actuators. These are sources or sinks of data that are input/output t0 the data integration and analysis apps running on the Foggers/Farmers. Because the Difuon Fog is an IP centric network, communication with loT devices will be over IP. Non-IP loT devices will need to communicate through gateways (not shown).The Relay nodes can help establish and maintain connections to the 10T devices and gateways.A full network with the

ST P Ea e, S NI S ASAS i Y,

ST <L " S - = 7 IS I X T i 0 Sl

et NN a /A\ ‘*.‘ f' \') N 2.8. Business Model The business opportunity is primarily leveraging blockchain technology, decentralized identity, cloud management software, 10 unleash their latent bandwidth, space, and energy with dedicated remotely managed servers at a near zero cost basis. This consumers, This combination is designed 1o drive two-sided network effects. The exponential two-sided network cllccts should business model combines the best of sharing economy and blockchain, Difuon's approach uses dedicated hardware that compute nodes because users can monetize spare bandwidth, space, and energy. With our remote management technology these capacity in the fog with a near zero cost basis. In summary, decentralized public fog computing infrastructure monetizes spare unleashes latent capacity in the edge with two-sided network effects at near zero cost basis! The core revenue generators are as follows: + Fees on purchases from the application and software compenent store. of-Exchange (MoE) tokens work together to enhance positive feedback loops and positive cross-side network effects. i

‘Market and Technology Opportunity White Paper 15 up processing that minimizes latency and transport costs while enhancing privacy, security, and flexibility. Indeed it is IoT data analytics and anlysis that was the prime motivation for pursuing a deceniralized fog computing infrastructure in the first place. The network effect benefis of IoT can best be realized when previously isolated o resources are enabled to interoperate. The well know primary bariers 10 unleashing the beneficial network effects of 16T are concerns about securty, privacy, and agency. Duc to & paucity of corresponding solutions, the foT market delivers value well below its potential. A solution that only addresses one or two of these problematic barriers will be severely limited. Difuon provides a platform for federated ephemeral edge computing that viably solves all three problems of sccurity, privacy, and agency.

Finally any private cloud application that requires fevels of privacy and security that are not easily attained in a public cloud computing infrastructure might very well be betier suited for the Difuon public fog computing infrastructure. In order 10 be decentralized und privato and sceurc requires that the public fog use a diffusc trust perimeter-ss sccurity model. This model inherently provides mechanisns to enhance privacy, security, and control over applications and data whilsestill being hosted on public compute infrastructure,

This approach s disruptive o the much more expensive cost basis of large centralized data-center hosted cloud services. Difuon takes a truly comprehensive best practices approach to the transforming compute infrastructure from centralized siloed one-sided networks to truly decentralized distributed two-sided netwrks. Only the latter can scale beneficially enabling the broadest adoption and participation of both producers and consumers of compute services.

1. ConcLusion

‘The Difuon Fog exploits and integrates several emerging technologies to provide a decentralized computing infrastructure. for Intemet of Things (IoT), Industrial Internet of Things (IloT), Intemet of Tiny Things (IoT), and personal data applications at seale. The Difuon Fog provides a diffuse trust ephemeral computing frusted exeeution environment with end-to-end security (at motion and at rest) and modular privacy (off-chain interaction) that runs a distributed serverless framework (FaaS). This is essential enabling technology for a truly decentralized public fog computing

infrastructure.

The distributed nature of ToT means that processing and analysis scales better if aggregated in a bottom up hicrarchy from Tocal to global. The data integration layer needs to be universal, not fragmented as it is now. It needs to leverage as much as possible the existing cloud based tooling for distributed applications but adapted to the unique characteristis of the fog.

Given the rampant fragmentation, stratification and siloization of existing ToT device level protocols, frameworks, and platforms, the only viable solution is an Intemet Protocol (IF) centric approach that enables reliable, secure, decentralized integration across and through ToT silos and strata. This solution is a decentralized federated cphemeral edge computing frameswork, platform, and marketplace for data integration and analysis. This s the Difuon Fog,

Samuel M. Smith Ph.D. Ts the CTO and co-founder of Difuon, a startup that is building a decentralized fog computing platform. He is advisory architect for reputation and Al for ConsenSys. He serves on the technical ‘govemance board of Sovrin.org. Samuel received a Ph.D. in Electrical and Computer Engincering from Brigham Young University in 1991 He then spent 10 years at Florida Atlantic University, eventually reaching full professor satus. His rescarch specaltics includc automated reasoning, machine leaming, and autonomous vehicle systems. He has over 100 referced publications in thes arcas and was principal investigator on mumerous federally funded rescarch projects. Dr. Smith has been an active participant in open standrds development for networking protocols and is a serial enireprencur: REFERENCES 174Alut SONM. Hipsidoss o om 21A. Back, M. Corllo T Dshie t a, “Enabing lockehsin Tonovations withPegged Sdechsins,” Blocksrar, s blockstean onssidecuiza i 1314 Ophr, A. o . Onca ta. I Rise of e D Loomomy: D Vaue tirughIteroc of Things Dt Monetiznton.” L1 Tech Repor, 2016 s public i o comcommontsiecm e o123 s lobal-businesssevices-coliv.solons - paper-<xeral- w236 Tusen 2016212 AL oy, “Thecloud computing ket s sbot 10 gt ot o commpetive” oo 20180214 s infowurd comaile 3255425l compating e-co-cormputin: ket -doul-o-g6t--lotore <ompein il 67 e, “What i 0 Decenisied Applicaion?,” Coindesk, s coindesk comsnfurmatin bil-a-Jsentrle-pplicaion-pp {61 W, roup, “OpenFog Arciecire Overview.” ol OFFWPO01 0316, 2016 s penfogcomsrtium o e ploads OperFog Arcifear e Wr.2.2016 it 71T Aublin, .. Mokhiae and V. Quéna, “Rb Rehndanthyzan

ine Faul olerance,” v Distibuted Computng Systerns, o, ICDCS, pp. 297-306, 2013 S1A. Asayag, . Cohen, 1. Grayevsky el “Flix: A Scalble a1 Fair Consensus Algorthm Resisant 0 Onlering Maniputon” Ors.com, g o comp-comentuploady 31 KOS N 13 pdt 91°AWS Serveles Appicaton Madel,” Amason Lanci, 101 Arure T Fage,” Micrso, sz microsol comensiservict i {118 B, “An Eficien B Digial Sgmatre Protocol Bused on Ellpic Curse” Imernasonal Jounal of Tecology Enbancemens and Emerging Enginering Research, w2 s , . 6275, 2014 1215, Vi, “Wht I Fo Compuing? And Why It Mters In O i Dt And 6T Word,”Fores, 2016110114

‘Market and Technology Opportunity White Paper 16 s forhos o bemardarr 20161011 4wt i-fog:computing-snd-why- i maters-n-ourbig o vl BSSober (3] “Bicoin” Bicoinorg. Inpsibicoin.orgienideseloper-documentaton 11473, Butler, “What s T computing? Comecting e cloud 10 hings™ Nesvork Wortd, 201810717 g networkwod comaicle 2431 iemet o ings/wh s o computing comneing the<loud 1 hings il 115 Bycamtin sl wlesnce,” Wikipci, Hatpsen wikipeliaorgodkiByzantne ft_werance 11617, Camenisch, “Coypuogrphic e Cash” 2016 npbicuinschool prdesiesions pdf. 171 Coles, “Cloud Market in 2018 and Prdictions for 2021.” Sy Highepworks, 2015 TISTM.Corke, “Daatlow and Reacive Programming Systems,” Create Space Independens Publising Pltfrn, 2014 191 Schmder, “Marrying Fphemers] Docker Containrs o Fersstent Dat,” i o, 201 6109715 s inue comnewemaryin sphemera dockercontiners st dta 1201 M. Castro and B, Liskov,“Practcal Byantine falt toleance. vl OSDI 99, 173156, 1999 211, Vil “Aversgo ntemer sposdsrose 23 percent st yes,” BGR, 2011710, i o 201RIT 1 feversge. et sposds-usvs e el 2], Clemeat . L. Wang. L. Alviss t ol “Making Byzunine Pl Tolerunt Systems Tolerte Byzanine ks, vol NSDI 9, pp. 153-165, 2009 23] “Cloud T e Goagl, psilond google comforedgel [24]C.5 Alance,“Custom Applicaons ad S Trends,” Mefee Repor, 20180401 [25] "CoAP: RFC 7252 Consisined Agplication Protocl, Conpfechnology, ipscnp echnslozy 126]C. LeMabien, “RaBlocks: A Feeles Ditibuted Cryptorency Network” Node RasBlocks, Tnpsiaiblocks netmedia RaiBlocks Whilepaper_English pdt [27] C. Cotia A 1. Fervinder-Leisa, F_F. de Vega et al, “Fphemeral computing snd bioinspired optmization: Challenges and opporunities” v Computatonsl Tteligence, o 1ICCT, pp. 319324, 2015 28] "Crowd Compurer,” ipsieserowdmaching comcrowdeomput 1291, Aol nd D. Hermandex, “Al und Compute” Opend! Bog, 20185115 ipsiblog.openaicomar-nd-compute 1301 R. K Das, 5. K. Nayak, S. K. Bho e al. “RSEA: A Bind Saloc-Bid - Avetion Scheme for F-Commerce Applicaions,” Computers, vol. 5, v 4, pp. 2, 206 (341D, . Stewarl, “Software Commponcatsfor Resl Tine,” Embeddd Systems Programing, 20001201 [32]D. Meres, “The Selae Comsensus Prtocol: ATederted Model for Inernetlevel Consensi,” 20160226 s stelaongpapersstlle <onsensus-prtocol il [397D. Schwartz, N Youne and A, Brt, “The Ripple Consenss Algortin” Hipspple o e rippl. consensas wiiepaper it [ “Decenialied Wentifies (DIDw)” WIC D Commanity Group Report 23 At 2015, Hipsic g g odid-spec 35 “Delegated Pronfor Stake Comsonas,”

BiShares, T bishars rgechnology deegatedproof of stk consenss 1361 1. Diar-Monias, Y. Xi, 1. Rodorn et o, “Federated Camputing (o the Masses.-Aggregating, Resovrees o Tackle Large Sele Fngincering Prblems,” Computingin Sieace & Enginsering ol 16, 0.4, pp. 6212, 014 1371, Stama, “The Fphemera Life af Dockorzed Mieroseviess,” ran, 201501122 g blog.ron.othe-ephemers-ife-of docksized 1361 “Ede ToT Anshyic,” 14, Hipscomsl bluemix.nedocsserviees 1T edge_analtcs e analytics 391 1L Kinan o - Amounas,“Proposd Developments of Bl Signatre Scher basd un The Eliptic Curve Discrte Lot Problen” Compter Engincering ond Applcions Jourul ol 2, n0. 1, pp. 151160, 2013 OV, Bochman, 1 Kwon and 2. Miloseic, “The Iest gossipon BET consensus,” 20117713 npstendermint comsdocs endenmint - 4 “ibereum.” Eihereun Fousaton, T thdocs ongenTaestinroductonndex bl 21 FCC, “Sinih Interational Broadband Data Repor,” 20 K022 npsansition e e Daly_Releases Dy Business 201 S0bOSDA.15.99A 1 pl 97 . Feldman nd . Micali, “An opimal probailisic protocol for snchronos Byantine greement” SUAM Jowowal on Compusing, vol. 26, 1. 4, pp. §739%3, 1997 ] o based pogramning,” Wikipedi, psfenwikipedisorgwikiFlowbased_prograoming 451 “Fog Computng,” GpenFog Consorium, s apenfogeonsortium.org what-we-daHdefinkion-of-fog-somputing 461 “Fog computing for industis automtion” Consaling Specifig Enginesr, 20 1R0318 oo esemsg comisinle-aicle o computing. Lo ndustcil-ormaion bl 7). Rescarch, “Developing Framework o Improve CiialIfsicucire Cybersecuriy” MIST, 0130405 e mistgovisite deflRles documents 201060 040813. ot rescarch pdl 4516, G Bdelan, “Alpowesed ambientcompulin i st gelling sated,”Fenrear, 201103730 Haipsenturcbea cory201 03301 pwered. et computing s just. et st 497 “Gelin staried with Edge Analytics in Waison 0T Pltforn.” 5, 30105/ 5 nipsideloper fom comirecipes il geting-saried wilh e analyis-in-wtson ol plaformy Tem_pnc uid- 969369926983 IAAITSIOcm._me_id SO200000-43260401 537208012579 SOTY. Gad, R Horm, 5. Mol “Algorand: Safing byantine gseements fo cypiacuencies,”vol. rocsedings ofthe 26th Symposium on Operating Systerns Prnciles, pp. 5165, 2017 517 Gilman and D, Barth, “Zero Trust Networks: Bing Secure Systems n Unirusted Nesworks,” O'Refly Modis, Tne", 2017 1521 “Gresngrass,” Amazon A5, s amazon somigrecngrass

Market and Technology Opportunity White Paper 17 15311 G, “The Ricardan Conact,” ysiemics, alpang oapaperscanian ot b 5471 G, “Trple ey Accountie.”Ssiemics, 2005 i ang oripapersiciple enty il 15571 Samiadoh, A brefnvodietion o o dats processing arhiectres L and Kappa for B Date” s Dot Science, 201 K03/13 npnadsdatasiencs com riel ntodetion-fo-oo-datprocesing arhiecure tan-and Kappa-for i dta 4T3SC2S00SH 15611 Lunden, “TAM o by Red Hat for S35 i cash and b, king biger o nto hybrid cloud,” TechCrunck, 201 1028 ipsitacheruneh,om;201510:IR bt buy-ted it oMb - cash-and-deb-aking:a-higer ety sl utn_medium=TChevwsletir 15712.. Morrkon, Pl based Programiming,pasimarison con, o pamorson com 1551, Sandors and C. Foret, “Hybrd cloud: What ¢, why it matiers” ZDNe, 20110701 s et comatleyrid-<loud-what s -t maters 15913, Ry, “Amazon's massive AWS outage was causod by human et e, 01703102 s oo et 2017537214 T92436/Amazon: - IemEL. vl <o aman-corngomsetcommand. 1601, Engebretson, “\GR: Avrage Monthly Broadband Usage 190 Gigabyts Montly Per Hovschold” TeieComperior, 201609:26 e <locompetorcom -8z month-broadband-age-c 190 igabytes-monthly-pr-houschold 611, Whit, “Privte v, Public Cloud: What'sthe Diffsence,” Pt 2015 s, expedient.comlogiprivte-vspulic-cloudwhats.difersnce 162] “Join s i uiing, an open soures decontlized detity ccosysem fo poopls, organizstions, apps, and device.” Decentraized deoty Foundation o, nptadenciy fundation 16313, Band,D. Farey, ._Kub ot af, “The Reaeive Manifest,”rcctsemanisi o, 201409716 s racivemanifesto o 16412, Boner and V. Klang, “Reetiv programming v3. Resstve systems,” 0 Rely, 201611202 s/ nsy comdea reactve programing 3 Tectve e 6513, Tidulk, “The 10 Biggest Cloud uages OF 2018 (So Far)” CAN, 0130801 s con comlde-showssecui 300107391 - 10-biggotcloud-ouages-of 2015 o-fr e 161 “ourmalof MyDeta, "y Dataournal ediem, 16713 oroscik, T fureofcloud comptin rlatie o becoming & monopol” Hog, 201710929 s 0. com 201 7109129 th-Suturef-cloud-compusng eltve o becoming-t-monopoly 16811\ Wagenen, "Connection Spec o Every Stac: 2018, St ech, 2013105108 Itttz comsriele 201 U5 aersge-niemetconncton-spesd-ofevery-sate 2018 perfcon 1691 "Kineie dgs” ipor, g aporis 701K Malws, Wy Reactve,” 0 Reil, 2017 s rily.comprogramming e why.feacive spTinemp=il v cbops o produce- it reactive programming v reacive systams (Xt ca [71/ L. E. B M. 0. N. BAIRD, “THE SWIRLDS HASHGRAPH CONSENSUS ALGORITHM: FAIR, FAST, BYZANTINE FAULT TOLERANCE 2m6051 e v condnloads SWIRLDS-TR-

2016-01pd 172] L Tung,“Github vals i from Microsof:acquision but ' o oass exodus, v Zde, 201806106 73] Columbus, “Roundup O Cloud Computing Forecusts, 2017 Forbes, 20130420 [74]L Columbus, “Forreste’s 10 Clud Computing Predicions For 2018 Forbes 20171107 Tpson: frbos comisloscolumbus 017 LT Toostrs-L0-clo-sompusng predietions-for 201413555713 [75] b Mansous, A V. Sumbra, S Hawke et al., . demonstation of the slid ltfom for scial web appliction,” vol. Proceedings of the 25 ntermational Confernce Compusion on Word Wide Web, pp 223226, 2016 [76] M. Kutskova, “MyData: uls and bolis" M 1 Dt Journal, s, comims daaimydata-utsd-bols (02T 437 17712 Mt and L Alvisi, “Fast byzanin consensus,” IEEE Transaerions on Dependable and Secure Computng, vol. 3, p. 202215, 2006 78] M. Shackler, “Edge Camputing: & cheat shect” TeohRepublie, 01707121 g ccheepublc comatcedgs computing the-smart prsons el 1791 . Marz and 1. Warse, "Bz Dts: Pincples andbest prscic of bl real-ts dat sstrns.” Neor York: Masring Publications Co. 2015 130] M. Asey, “The futare f sererlsscloud ook ot ik physical srvrs ™ TochRepubic, 201710829 ips oo nepublic. ot . utre-of-srveres-cloud-ooks--ot-bke. physcal-seven 811 M.G. Zago, "Why the Web 3.0 Maticrs und you should koo about " Medium, 01501730 s o 09wy A b-3-0-les-an oo s 1SS 146349 52) M. Kunelo, 152,000 S Devices very Minu I 2035: C Outlines The Fulur of Sman Tnings.” Forbes, 201603103 53] “Microservices,” Wikipedi, lapsenwikipedinrgiviki Miczoservies 847V e, “Cond: A disibuted ledge” cord.nc, 2016711129 e conlnedcomentconl echrical-wbitepaper BSTM.LT. M. Lab, “The T Dt Taarami Tas Arved, Now What?” Conncied Things 201300405 s connectedhings miforumeambridee oroversiewagendi-comecied hings 2018l dat i 1861 My Data.” Mo Doto2015, . ipsmydata20 8 ore 187) O C2. HaniKernl, g barikernel o mainproductsprl_inlopbp?e_code-GLASAST2INE 88 “Open Porable Trustd Exccution Lavirnmeat,” OP-12E.org, ipsewop-cen: 8910, Consorti, “og Computing.” g openfogeonsorumongresources 90] P Brogam, “USTeleum Indusiey Meccs and Tres 2018.” USTELECOM, 2018103101 Tt uselecom onsites deiul lesages US Telcom 42 sy 20Melris620s%20 Trends 202018 . 9177 Brogan, “New Aalysis: USTelecon Data Highighs U, Broadband Epansion, Competiion and Usage,” USTELECOM, 201813720 T ustelecom g blog/new analyss stelecom i highighis s broadhand.expansion-competiion 41 e 1921 Marrison, “Flows Tosed Pogramming,” fomhasedpogramming con, 201 204115 Tnpsilowbssedprogramming wordvess.comianiclTow. besed pro

gramming!

‘Market and Technology Opportunity White Paper 18 93] “Personal Cloud.” Wikipedia, 54 Bublies,“Duta i iving i 03w ccomutny: Hovw is i shpiog up?,” The Economis, 201105/06 Ipsanenecomomist combriefng 201 T0S/06at--goving. e o-i-new-sconmy 95 “Plemum Byzantine Faul Tolerant Proocol” HyperLedeer s, it comhyperiedgertndy plenmiviki 961 ot orwadine.” Wikipecia, npsienwikipediaorg/wikiTTon_forasding 97 Proutlsiaker Wikipedio, npsen wiipadisorguikiTrout of ke 98] “Proatar work system.” Wikpedia, npsen wikipedisorgaikiTronf of sork_sstem 1991 “RandomzedConsenses Ve, ks ysle.cdhomessspnes pinewiiRandomizedConsenss 1100] “Raspberry P Model B+, Rasherr, i, e sphernyp g produsraspherry-pi-+-mods- pus 1011 .G, Brown, “The Carda Platform: An Intadicton,” Cord et 201415001 ipsnen conb edcomieny ol la for wilepper . [102] Rockss.” Piness, s pineS o page id-7147 [103] T Rocket, “Smowliske o avalnche: A ovel melasabie comsenss protoce fmily or cryplocurencies,”” 2018 [104] R Miller, “Eds computing coldpsh th cloud e [ige.” TechCrunch, 010803 Hipstechcrunch com2017/0810 e computing could-pus he-coud -t Finge! 110515, M.Smith, “Toflo Tl com, it comfofoioflo 10615, M. Smih,“Toflo Documentaion,” ot com, it comfiofliollo marials 10715, M. Smih,“Relabl Asynchronous Fvent Tanspor (RAFT) Protocol” RaetProcol, st com RastProtocalinet 10R] 5. M. Smih, “Open Repataion Fameveark” vol. Verson 12, 2015105713 gt comsSethSamuel M/ lohnaster hitepaperapen-eputtion o levl-whitepapspdf 10915 M. Smih, “Niany Culbed: Slvin th Many: <o Mans-1o-Many Problen for Scable Relible Socure Ditibused nernet Apliadons” 2017101105 st com St Samuelh Papers lobmaser whicpapérsManyCabed - [110] . M. Smith, “Decentlze Autononi Dals (DAD) ad the e K's uf Key Managernent,” Sprig 2018 it comAVebOFTeustIf'ebootin .- us spring2018 o st Bl documals Desentized utonemic Dt pll. [111)S M. Smith s D, Khovrtovih, “entity Ssten Essntisls.” 201603729 [112]. Nakammots, “Bitoin: A Pecea-Peer Eeconic Cash Sysem,” 200811031 aipsbitcyinoxghicon pll TH137S. Fllon, “Is rae o h e, the end of v computing s we know i ZDNer 2007111114 Tnpnow: et comfarice i ace o the. edge .- end-of clod computing: s e know i 1141, Popos, “The Tangler” o ors, npiatoken conTOTA. Whitepapepdf 115 Serverlss Compatin,” Wikiedio, npson wikipadisorgikiServales computing T116]S. i, “Updat: 11 hour AWS il is ebsis and spps,” Compterworld, 2010225 e computosorkd comailes TS643 o comping/avs falure s wob-sics and-apps Ml T11T]S. Consay A H, M. Mt a, “A DID f

or Eerything,” RIFOT Fall 2015, 201809726 it comNEhOTrust oot 18]S, Cares and N, Randevich, “Amason touches S nlon, o pace o overake Aple” Restes, ol Businss Novs, 201809104 g reutens comiatclefos-usa slocks- o com oz ouchesAlion-dn-pace-o-overake apple-AUSKCNILK 12 [119) “Solwsredefined networking Wikipedi, Hatpsenwikipediaorfoiki Software define_networing [120] “SOld Esphined, solid nrupt com, Hipssold nuptcombore i vorks 1217, Chirara, “A quick ity g 1 it computin (snd wh i winni s0 )" The Ambicns, 201801104 e the-ambient comyeturesickand dily-guide ambint computing 21011210 11221, Foundation, “Sovin: ey For AlL” o, 1251 Sovri: A Prokocol and Token for Self. Soveeign deniy nd Decentlized s Svrin ong, 201501101 ipsiowin.arghup-contatplosds20 /03 Sovin Proocol-and-Token-White D pf 1124] “Spesdiet Global Tndex,” e, 01510501 o sposdiet neglobalinde 11251, Cli,“Triedgor in Plain Englit,” ipfiredgercomidocplan-englh bl 11261 Statisics, “Viarke spializaton o th biggest et companics worldwids s o May 201 i ilon U, dollr)” Salstc, oot comsatises 217483 ket vlue-o e ettt commpanies-wordide [127] M. Steemstcup, M. A A E. G. Maes, “Port atomata nd the algebes ufconcurrat peoceses,” Journal of Comper and System Sciences. vol. 27, D01, pp 20.50, 1962 T126] . B Sewan, “Designing software compunentsfr e pplicatons.” vl Proceedings of Embedde Sysem Canlirence, 2000 T129] M. Swan, “Blockeluin: lacprintfor e ecumomy.” - O'Reily Medi, I, 2015, [130] “TCP ok punching,” Ftipedia, npsen wikipedisory i TCT._bale_purching, 1311 “The Fiach Algoritn,” Eferam Foundoton, i thdocs ongenTatesimining Wb algortin [132] “The Futur of Compute: A Gloal Make” Acie esr, Hipsctvenethrcom 11731 Rittan, “The Edge Wil Et th Cloud” Garter Maverick Rescarh, 2017109722 s gannercomdos eprintsTid—1-4GHOFTL &t 17100485650

‘Market and Technology Opportunity White Paper 19 1361 A& Th K. My, “Implemenaton o ancicine ind st chem,”teratonal Joul of ovain, Maxgementad Tchnoiogs, vol 206,95 443,2014. {1391 T onke, M. Movabeds and D, Wi, “DFINITY Technlogy Overiew SeisCoscasas Sysiem:” gty g viowerpsvione e Tbrar ity conscms il 136] “Trle Signed eceipts.” OpenTransaciions. o, g peniamsctons okl phile Trple-Siped Receipts (137 UDP bl uncing” M, inpsin wikpelivorgwHGUDE hole pncting 1381 e, T Digl Uit o Opporaies Rich Dot e Tcrssing Vel of e nene o Tings.*IDC Rport, 20140401 391 Turer, il O Senors king Millon O Ouicones” 21 G084 ok aowcamar-st downlodsipdFdownload e o ofsensors g 1401 uthor 34 € MG, TDC FutaeScape Worlolde 0T 208 Prsicions, IDC Reors, 20171101 s decomigetTle dyTcciond-ISTOFATTACHMENTS coninela- US 1193617 clementa-US 319361 - D000 Satachmend 47552915 V411 B, “The Visaing f Decontliztion; M, 20170217 e o @B e o docnlion s0e9 2164274 {1421 ot V. G, “Cospe e rcadly sty ol gt comlbremesesr O e ppers spe bsicscsper i T {145] Woho, Rication ot n OpenBar G, [14] “WILAT 1S A ZERO TRUST ARCHITEC TURE?,”Cyberpedin, s paloalonevrks comiyberpeaht - s s achiectre 145] Whipperos” gtk ngibi Whicpaper ot {146] Wikipedis, Fog Computing.” Fikpeds, g en kel o, compting 147) Wikipedi, Mot Digia R Wi g e palisorg Wi il 148] ke brodbind speedagx 015 e sl eosdhand spesdri dvid.spoc s gions 14917 Wirspoon, A HHNkers Guid Consenus Algdthms” Hckeroon, 20180213 peackemoon oy MG gide.t-consnsu sl 4 c3ehoes 1501 Tanoknwldgo proot? Wik, psen wpeliorgii e Knovlede el