Setting up btcd + Go build for bitcoin

My last post was about setting up the build system for Bitcoin reference system 0.9.0.

There is an alternative architecture for Bitcoin called btcd which is developed by Conformal Systems. This is claimed to be compatible with the main blockchain (including bugs).

There is a very interesting thread about how the btcd architecture (especially the split wallet/client and daemon architecture) has been adopted in the reference client at 0.9.0

I find Go very, very pleasant and productive to work and understand and it’s package manager is absolutely brilliant.

To setup your machine to work with btcd is absolutely trivial. Remember that this should be the same on any platform (Windows, Linux and Mac) since Go is cross platform in general. Only the particular binary of Go would be different.

Download and unpack Go from http://code.google.com/p/go/downloads/list . I used http://code.google.com/p/go/downloads/detail?name=go1.2.1.linux-amd64.tar.gz&can=2&q= because I’m on Linux-64 bit but go ahead and use the one that you’re on.

Assuming you unzip it to /home/sss/Code/go, set the following variable:

export GOROOT=/home/sss/Code/go

Test your Go installation by running /home/sss/Code/go -v . Ideally this environment variable should be in your zshrc, bashrc, etc. This never changes.

Now, create a directory called /home/sss/Code/mybtcd. This is your new workspace. When you are working on a particular workspace, set the following environment variable:

export GOPATH=/home/sss/Code/mybtcd

This tells your Go package manager, the location of your top level workspace directory.

Now, to get btcd and all its dependencies as well as compile it in one shot, run:

/home/sss/Code/go/bin/go get github.com/conformal/btcd/…

After a few minutes, you should have the following directories (which complies with Go’s recommended workspace directory structure)

./bin/ -> all your binaries

./pkg/ -> all third party library dependencies

./src/ -> all btcd as well as dependent third party source.

Running your bitcoin daemon is simply ./bin/btcd (help is at ./bin/btcd –help)

To hack your code, just write your code in ./src/github.com/conformal/btcd/ and run

~/Code/go/bin/go install -v -x  github.com/conformal/btcd/

All dependencies and binaries get rebuilt. Simple.

Compiling bitcoin 0.9.0 – the transaction malleability fix

I had a bit of trouble compiling Bitcoin 0.9.0 (which contains the all important “transaction malleability” fix). So I’m posting here for the benefit of everyone. This is done on an Ubuntu 12.04 machine – which is relevant only for the system packages (if you’re on any other machine just ask around for what are the equivalent packages)

git clone https://github.com/bitcoin/bitcoin.git

cd bitcoin

git checkout v0.9.0

sudo apt-get install build-essential libboost-all-dev automake libtool autoconf #this part is dependent on Ubuntu

mkdir $PWD/release #I dont want to install bitcoin systemwide, so I make a local dir.

./autogen.sh && ./configure –with-incompatible-bdb –prefix=$PWD/release

make

make install

 

P.S. if anybody reading this is on another platform and figures out a slightly different way of doing things, please post in the comments

The intricacies of Bitcoin

What are some of the under-the-surface aspects of Bitcoin that safeguard its larger application as a viable digital currency

transaction fees

each transaction in Bitcoin is subject to transaction fees, that prevent something called Dust Spam. Where do these fees go ? Txn fees goes to whoever processes the block that contains the transaction. Additional reward for miners. Very cool !

whichever miner solves the next block gets to include a transaction for 50 BTC to themself from “coinbase.”  It turns out that  that if any of the transactions the miner included in your block had a fee attached to it, then the miner gets to include those, too.  Therefore, when a miner solves a block, it typically gets something like 50.75 BTC instead of 50.  The more transactions there were, the more fees received.

If you look at the BTC webpage description about what happens when there’s no more rewards for solving blocks, it mentions that they expect the network to be big enough by then that it will be worth solving blocks solely for the fees.  If there are 10,000 transactions per block, at 0.005 per transaction fee, that’s 50 BTC in fees.  If BTC really catches on, this is a realistic volume of transactions

transactions fees are also voluntary – a transaction fee will increase the chances that a miner will include your transaction in the block he mines. Actually, a miner  just dump the top few hundred KB of  transactions into a block, sorted by transaction fee (descending, of course). When there aren’t many transactions, maybe because of a series of block in a short amount of time, it will be confirmed anyway.

Do Transactions get lost ?

When you send a transaction, it sends a packet to all connected peers. These peers store the transaction in their in-memory pools and tell all their connections that they have a new transaction. When those connections don’t have it yet, they ask for it, and that’s how a transaction spreads over the network.

When a user restarts their client, the memory pool is wiped, and the unconfirmed/unmined transaction is deleted from that computer. But it is still available on other clients. it’s very unlikely for the transaction to be gone from the entire network.

Solving a block

a block is a list of transactions broadcast by the Bitcoin network. This system evolved because of the question “how do I build a distributed transaction network without a central authority“. What will motivate people to contribute computational and network time to the Bitcoin system ? Well, the chance to make money.

Bitcoin miners act as distributed banks – or more aptly, those irritating credit card salesman who tell you “please take this credit card“. Each of them is trying to be the eager salesman and be the first to “process” your transaction – and the way they do it is solve a puzzle. The process of “Mining” is essentially the process of competing to be the next to find the answer that “solves” the current block

The mathematical problem in each block is difficult to solve, but once a valid solution is found, it is very easy for the rest of the network to confirm that the solution is correct.

Miners are essentially putting a notary stamp on a batch of transactions. That’s all they are needed for.

But how do you prevent from having a corrupt notary? Bitcoin does this by having tens of thousands of potential notaries and one of them will happen to be the lucky one that gets to do the stamp. The lucky one is the one who happens to solve the problem. All the potential notaries try to solve the puzzle over and over, but it will take about ten minutes for one to become successful.

Difficulty

Essentially it is a cost function that determines how hard hashing should be so that one block is found every 10 minutes, on average.

Pool difficulty 1 is 0x00000000FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF

For every 2016 blocks that are found, the timestamps of the blocks are compared to find out how much time it took to find 2016 blocks, call it T. We want 2016 blocks to take 2 weeks, so if T is different, we multiply the difficulty by (2 weeks / T) – this way, if the hashrate continues the way it was, it will now take 2 weeks to find 2016 blocks.

P.S. 2016 blocks in 14 days = 144 blocks per day. This is expected difficulty

The difficulty can increase or decrease depending on whether it took less or more than 2 weeks to find 2016 blocks. Generally, the difficulty will decrease after the network hashrate drops.

If the correction factor is greater than 4 (or less than 1/4), then 4 or 1/4 are used instead, to prevent the change to be too abrupt.

NOTE: There is a bug in the implementation, due to which the calculation is based on the time to find the last 2015 blocks rather than 2016. Fixing it would require a hard fork and is thus deferred for now.

Difficulty should settle around the 70-billion mark, assuming 300 USD/BTC, 0.08 USD/kWh, 1J/GH (with Gen2 ASICs dominating the field).

A bitcoin miner’s profit relates to the amount of hashing power they contribute to the network. Since their mining power is constant, their share of the total hashing power decreases relatively when the network’s hashing power increases. I.e.

newProfit = currentProfit * currentDiff/newDiff.

At a currentProfit of 1BTC/d and a 30% increase in difficulty, they get:

(1BTC/d)*100/(100+30)= (1BTC/d)/1.3 = 0.76923077 BTC/d

i.e. their profit decreases by ~23%.

Network Hashrate – the mathematics of difficulty.

To find a block, the hash must be less than the target. The hash is effectively a random number between 0 and 2**256-1.

hash_rate = (blocks_found/expected_blocks*difficulty * 2**32 / 600)

Block maturation

Generated coins can’t be spent until the generation transaction has 101 confirmations. Transactions that try to spend generated coins before this will be rejected

The purpose is to prevent a form of transaction reversal (most commonly associated with “double spends”) if the block is orphaned. If a block is orphaned the coinbase reward(block subsidy + tx fees). “ceases to exist”. The coins are produced from the block and when a block is orphaned it is the replacement blocks version of the coinbase tx which is considered valid by the network.

So to avoid that undesirable situation the network requires coinbase tx (rewards to miners) to “mature” or wait 100 confirmations (the client makes this 120 confirmations but only 100 is required by the protocol). If a block is orphaned before it gets 100 blocks deep into the chain, then only the miner is affected.

The referece code that checks this is

 // If prev is coinbase, check that it's matured
            if (txPrev.IsCoinBase())
                for (CBlockIndex* pindex = pindexBlock; pindex && pindexBlock->nHeight - pindex->nHeight < COINBASE_MATURITY; pindex = pindex->pprev)
                    if (pindex->nBlockPos == txindex.pos.nBlockPos && pindex->nFile == txindex.pos.nFile)
                        return error("ConnectInputs() : tried to spend coinbase at depth %d", pindexBlock->nHeight - pindex->nHeight);

Block Size Limit

Currently, the block subsidy reduces the motivation of miners to include transactions, because 99% of their income comes from the subsidy. Including zero transactions wouldn’t impact them greatly.

But when the block reward shrinks, miners may get 99% of their income from transactions. This means they will be motivated to pack as many transactions as possible into their blocks. They will receive the fees, and the rest of the bitcoin network will be burdened with the massive block they created. Without a hard limit on block size, miners will have incentive to include each and every fee-carrying transaction that will fit.

So a single entity benefits, and everyone else shoulders the cost with very little benefit.

 

Blockchain fork and double spending

If you understood what a block is, you can ask the question – what happens if two independent miners find an independent answer to the puzzle of “solving a block”?

Good question and this is called a blockchain fork – because now there are two ways everyone else can accept what the “current block” is. The way this is resolved is that The client accepts the ‘longest’ chain of blocks as valid. The ‘length’ of the entire block chain refers to the chain with the most combined difficulty, not the one with the most blocks. This prevents someone from forking the chain and creating a large number of low-difficulty blocks, and having it accepted by the network as ‘longest’.

Now, remember that the miners decide which chain is valid by continuing to add blocks to it. The longest block chain is viewed as the valid block chain, because the majority of the network computation is assumed not to come from malicious users.

- Race conditions

If you wallet accepts incoming peer connections, they can potentially control the information you receive (by flooding your connections). This means that it is possible to convince you about transactions that the larger network is rejecting. So, if you are a Bitcoin accepting merchant, disable your incoming connections and connect to reputed nodes to confirm transactions.

It is worth noting that a successful attack costs the attacker one block – they need to ‘sacrifice’ a block by not broadcasting it, and instead relaying it only to the attacked node.

There is a variant of this called the Finney attack which needs the collusion of a large miner – unlikely, but still possible.

- Brute force and >50% attacks

The attacker submits to the merchant/network a transaction which pays the merchant, while privately mining a blockchain fork in which a double-spending transaction is included instead. After waiting for n confirmations, the merchant sends the product. If the attacker happened to find more than n blocks at this point, he releases his fork and regains his coins; otherwise, he can try to continue extending his fork with the hope of being able to catch up with the network. If he never manages to do this, the attack fails and the payment to the merchant will go through.

NOTE: Theoretically someone with massive computational power can create start controlling this, but that is highly unlikely.

Aptitude equivalent of “apt-get autoremove”

As I am moving away from Ubuntu towards base debian, I am realigning my workflow to work better in the latter. The biggest being, unsuprisingly, package management.

One of the biggest cons against aptitude was that it does not do “autoremove” as apt-get does. This actually turned out to be false and more of a planned design choice (read more here – http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=683866)

In essence the aptitude equivalent is “sudo aptitude -oAptitude::Delete-Unused=1 install”

Getting a fully activated Windows 8.1 installed on VirtualBox – and the crap that Microsoft puts you through

  • Create a new VM on VirtualBox. Make sure you are running atleast VirtualBox 4.2.16
  • run vboxmanage setextradata <name of VM>  VBoxInternal/CPUM/CMPXCHG16B 1
  • vboxmanage list vms - get the UUID for the VM that you just created
  • vboxmanage modifyvm <UUID of VM> –hardwareuuid <some random UUID> - what this step does is create a hardware uuid for your VM. Now, after you activate windows, you can clone this VM and not lose activation. (Dear Microsoft, this is not to screw you, but give myself the power to be destructive to VMs. I am using a VM, so that I dont have to be too careful with it.)
  • Download a Windows 8.1 ISO from http://winsupersite.com/windows-8/windows-81-tip-download-windows-81-iso-windows-8-product-key
  • Make sure your “network card” in VirtualBox is disconnected. This way you can make a local account, instead of being tied to a Live account. This has a lot of problems just in case you are not connected to the internet at a later stage of using your PC.
  • Use a “Windows 8.1 Generic Key” to install your Windows 8.1 ISO
  • Once Windows is installed, you will need to change the key to your key. To do that, Right-Click the Start button and start the Command Prompt (Admin).
    Type the following commands:
    slmgr /ipk YOUR-KEY-WITH-DASHES-HERE
    slmgr /rearm
    reboot you computer and it should be activated.
  • To get full screen on VirtualBox, you need to use VirtualBox Guest Additions 4.2.16 ISO (not a higher version – there is a bug)
  • For heavens sake uninstall “Skype for Windows 8″ tile and install proper Skype from here.

 

God – seriously ? These are all the steps I need to follow to install an OS that I have bought on VirtualBox ? This is bullcrap.

Microsoft, please learn to trust your customer base a little bit more.

Quick and dirty NoSQL cheatsheet

 Mongodb
data-model
  • document-oriented database – not key-value database
  • maximum value size is 16mb kept in the BSON binary format.
  • for any sharded or clustered setup, all reasonable queries must happen through map-reduce rather than the query framework.
performance:
  • biggest con is the database level write lock, which starts impacting fairly quickly as your data size grows.
  • must finely tune indexes for performance – all other queries must happen through map-reduce.
replication and clustering: Master/slave replication with datacenter level failover (http://docs.mongodb.org/manual/data-center-awareness/)
CAP tradeoffs:
  • MongoDB tunes for consistency over availability – via its locking and a single master for accepting writes.
  • There is no versioning.
consistency:
  • warning: default configuration does not acknowledge write and is the reason for its performance. on turning on acknowledged writes, performance  drops to be same or worse as other nosql systems.
Couchbase
data-model
  • document-oriented database – not key-value database
  • biggest pro – memcached compliant api.
  • You can store values up to 1 MB in memcached buckets and up to 20 MB in Couchbase buckets. Values can be any arbitrary binary data or it can be a JSON-encoded document.
Performance:
  • uses indexes called “views” for performance.

Replication and clustering:

  • master-slave replication. All writes must happen on the master.
 
CAP tradeoffs:
  • prefers consistency over availability. all writes go to single master.
  • biggest con: Has eventual persistence – Writes are *not* flushed to disk immediately for performance (kept in RAM).
consistency:
  • does not support transactions or MVCC. it is ACID compliant on a single operation, however eventual persistence means that data can still be lost.
CouchDB+BigCouch
data-modeldocument-oriented database – not key-value database
replication and clustering: master-master replication as well as the most seamless cluster setup among all nosql (bigcouch merged into couchdb)
performance:
  • CouchDB allows for creation of indexes on separate disks (SSD?) called “views” that speeds up queries by many orders of magnitude.
  • biggest con – only access is through an inbuilt REST api which adds about 100ms to each request. All data interchange is through JSON which means performance for large documents will suffer (due to serialization)
  • queries are implicitly map-reduce.
  • relies on page-cache for performance.
consistency: biggest pro is that it has built in MVCC and transactions, so there will *never* be race conditions between read and write.
CAP tradeoffs:
  • prefers availability to consistency – especially in its multi-datacenter setup. All checks and balances happen through revision numbers that means disk usage increases pretty fast (due to previous revision numbers).
  • needs periodic compaction to clean up.
Cassandra
data-model
  • key-value database   – not document-oriented database
  • 2GB column value. maximum number of cells in a single partition is 2 billion. however different partitions can be on different machines/vms.
replication and clustering:  Cassandra is aware of network topology and does cross-datacenter replication fairly robustly among all the nosql systems (http://www.datastax.com/docs/0.8/cluster_architecture/cluster_planning)
Performance:
  • Cassandra tends to sacrifice read performance in order to improve write performance. Because of the log structured design of Cassandra, a single row is spread across multiple sstables. Reading one row requires reading pieces from multiple sstables. However, this comes with a much, much higher fine tuning control on disk layout and data locality.
  • Cassandra also relies on OS page cache for caching the index entries.

Consistency:

  • Cassandra does not offer fully ACID-compliant transactions, however the cluster itself can audit success/failure. For example, if using a write consistency level of QUORUM with a replication factor of 3, Cassandra will send the write to 2 replicas. If the write fails on one of the replicas but succeeds on the other, Cassandra will report a write failure to the client. However, the write is not automatically rolled back on the other replica. However, your application does get to know the failure condition.
  • There are no locks
  • no MVCC – Cassandra uses timestamps to determine the most recent update to a column. The timestamp is provided by the client application. The latest timestamp always wins when requesting data, so if multiple client sessions update the same columns in a row concurrently, the most recent update is the one that will eventually persist.
CAP tradeoffs:
  • Cassandra supports tuning between availability and consistency, and always gives you partition tolerance. Cassandra can be tuned to give you strong consistency in the CAP sense where data is made consistent across all the nodes in a distributed database cluster. A user can pick and choose on a per operation basis how many nodes must receive a DML command or respond to a SELECT query.
  • Writes in Cassandra are durable. All writes to a replica node are recorded both in memory and in a commit log before they are acknowledged as a success. If a crash or server failure occurs before the memory tables are flushed to disk, the commit log is replayed on restart to recover any lost writes.

Configuring the TP-Link TD-W8968ND for Airtel IPTV and Broadband

As we all know, Airtel no longer supports the equipment that it gives you. So if your modem gets spoilt you better buy a new one. You could buy the exact same one as they gave you, but it is much nicer to buy a much nicer one – with integrated wifi and modem capacity, as well as a capability to use EWAN (for Tikona) or 3G internet.

Which is why I went with the TP-Link 8968ND. I bought it for a fairly cheap 2300 rupees. Try to get Version 2 of the hardware which will enable you to have 5dbi antennas for increased range – however this router does have replaceable antennas, so you could upgrade them later on.

 

The screenshots below show the configuration that must be set in for the broadband and IPTV to work (make sure you connect the IPTV cable to the port that you have earmarked for IPTV).

8968ND - DSL settings

8968ND - LAN setup8968ND - IPTV config

8968ND- WAN config

SteamOS is here – my prediction came true

Almost a year back, I had predicted and requested that Valve consider building an entire distro on top of Linux, and with it the ability to armtwist nVidia and ATI to build great driver support in Linux.

Update (Sep 24, 2013): Well, that day is here – Valve just announced SteamOS – a completely new distro built on top of Linux, however with a few improvements.

In SteamOS, we have achieved significant performance increases in graphics processing, and we’re now targeting audio performance and reductions in input latency at the operating system level.

This is going to be huge in the Linux world – the base OS that Valve builds will undoubtedly come with its standardized libraries, driver support and third party hardware support. That could potentially be used a foundation to build a desktop OS as well that could well disrupt the market.

Oh, and the first casualty is already here. Welcome nVidia - they have begun to work with the Nouveau open source graphics drivers.

NVIDIA is releasing public documentation on certain aspects of our GPUs, with the intent to address areas that impact the out-of-the-box usability of NVIDIA GPUs with Nouveau. We intend to provide more documentation over time, and guidance in additional areas as we are able.

Platform wars: .NET and the JVM

A chance conversation with a friend got me to think about two of the most dominant platforms out there – .NET and JVM

note: I dont say Java or C#, because these platforms have given birth to more languages than the two most popular ones. Clojure, Scala, F#,etc. are some of them.

In particular, I want to think about whether there is a crippling effect to the overall strategic engineering roadmap if you choose the obvious devil – the .NET stack.

JSP vs Silverlight

JSP vs Silverlight is a case-study in the business of risk of adopting technologies that have a single point of failure in parent companies.

JSPs were invented by Sun in 1995-2000. It’s not the best technology today by any stretch of imagination, but it has been kept alive by users who choose to use/learn/teach them. Development of the JSP codebase has progressed over the past two decades as the Java platform evolved.

On the other hand Silverlight was originally released by Microsoft in 2007. It got its big win with the Beijing Olympics in 2008 (which used Silverlight for streaming).

Microsoft abandoned it in 2012 and took with it millions of dollars of technology investments by companies all over the world. It is a fundamental case study in the business continuity rewards of betting on true open source.

License costs

Here is a nice article by Microsoft SQL Server guru Brent Ozar on the various licensing cost issues that begin to hit you as soon as you graduate out of the Microsoft Accelerator program or start to handle any decent amount of computation/traffic.

Now, dont get me wrong – I do not mean to disparage SQL Server which is a seriously marvelous piece of technology (unrivaled by Mysql or Postgres till now). However the 98% of use cases that dont need the secret sauce of SQL Server, will be seriously crippled by the exponential license growth.

Why is this important ? Frequently, companies have a sunk investment into expensive technologies like SQL Server, and therefore the question begs to be asked – so what ? The problem is that as you refactor/rearchitect your system to a split service-based architecture, you will need independent self-contained teams/systems. Now, the volume licenses of a single enterprise SQL Server license is no longer relevant. I wouldnt want my teams productivity being crippled by sales negotiations into 3 new licenses.

And I dont even know the impact of these licenses on test and pre-production setups.

Rob Conery was a member of the ASP.NET team at Microsoft. When he started his company – Tekpub – he moved away to using Ruby on Rails and MongoDB. This is what he had to say.

RB: So what does the TekPub platform look like today?

RC: We moved to Rails 2.3.5 using MongoMapper against MongoDB. We have a reporting setup that uses MySQL to track stuff we need to report on which uses DataMapper. We also plugged in New Relic RPM to keep track of our site and it’s health – all of this is less than 1% of what it would have cost us, on average, with BizSpark.

JA: This solved all of our issues around licensing, we pay about $80/month for a large Ubuntu instance on Amazon EC2, after the reserved instance fee. It’s a technology that both Rob and I really enjoy working with, we have great testing with Cucumber, and deployments with Capistrano are simple and easy.

JA: All the problems we had (licensing, testing, deployments) could have been overcome. We could have used workarounds, written our own deployment framework, etc. What is comes down to is that we both really enjoy working with Rails and we enjoy the Rails community and the tools and libraries available. One of the best parts about running the show is that we get to do what makes us happy.

RB: How hard is it to do testing on a Microsoft stack and why?

RC: The language and the tools. If Microsoft thought more about it, it would be a bit easier to test the stuff. You can’t do RSpec with any language other than Ruby really – it just makes it easy. Microsoft could capitalize on that with the DLR… but they don’t. It’s not in their “.NET Story” and that’s OK – it’s their business decision. It was ours to move away from t

NOTE – I’ll be talking about Capistrano in the point about Impact on Service Oriented Architecture/Distributed Systems

Legal impact of licenses and patent coverage

Mono/Xamarin is basically an open source fork of the .NET runtime – Microsoft has actually seen it fit to encourage this development. However, there are certain aspects of the runtime that are not available on Mono (for legal or technical reasons).

Until the Oracle vs Google lawsuit is done with completely, the legal implications of both .Net/Mono and JVM platforms are a little doubtful. To a very large extent, this will not impact server-side applications, because the license talks chiefly about “redistributable” code – i.e. if you are shipping devices with these pieces of software on them (like Android was doing).

In general, server side applications are exempt from patents (from Oracle or Microsoft) being used to disable core functionality.

Superiority of technology

I would say it is an even match here. The .NET runtime and C# is quite well designed (by one of the greatest language designers on the planet – Anders Hejlsberg). They undoubtedly learned from some of the mistakes that Java did. The biggest con is obviously the fact that is tied down to the Windows platform (and its cost implications). In fact, C# (along with the Xamarin stack) is now one of the most effective ways to build apps for mobile devices across all platforms (iOS, Android and Windows Mobile)

On the other hand, the JVM is one of the most performant runtimes ever – very few things can come close to it in terms of performance. The Java language however is not the best designed language (when you compare it to C#). However, the rapid rate of language development on the JVM platform has given rise to some of the most advanced programming languages ever – Clojure, Scala, JRuby, etc.

Now, while F# is another language built on top of the .NET runtime, but it is not as production tested as some of the newer JVM languages – for e.g. Twitter (Scala), Twitter-Backtype (clojure), Factual (clojure).. and my very own Tradus (Jruby – atleast circa 2012).

NOTE: I later realized that, the core of LINQ essentially the same as the core of F# : two operators Return(x) and Bind(m,f). The name of Return is datatype dependent in LINQ, but its purpose is to create a container with a single thing inside.

This does not sound like a problem, but it is – because some of the most exciting technology platform of recent times is being built on these newer languages and .NET is simply not able to keep up, regardless of how superior the platform is – e.g. Storm (in clojure).

One of the big *rumors* against .NET – lack of systems for building web services is simply not true. For example, ServiceStack is an amazing piece of architecture with one of the fastest JSON parsing engines around (used by Stackoverflow, among other companies). There are many more than I would like to list, but many startups default to using ASP.NET MVC to build web services … which doesnt make sense.

Modern Hardware

Both the .NET framework and the JVM are equipped to deal with multicore on modern day hardware. However, there are certain technology fabrics (like Azul) which extend the power of JVM on grids and bring together really advanced tech pieces like Software Transactional Memory.

(more on this in the next section)

Community

“Productivity”, as is meant in modern day programming practice, is a function of leveraging existing building blocks. These are pieces of fundamental technology built and maintained by the community – which could be both the parent company, or by an open source community worldwide.

The “officially” sanctioned community repository for the .NET platform is Codeplex.

The JVM is absolutely unparalleled in this regard.

To dig deeper into this, I took a look at what is probably the current poster boy for .NET development – StackOverflow. Jeff Atwood (a highly experienced .NET programmer for a long time) made the decision to go with ASP.NET and has blogged about his justifications. One of the big things that .NET made is possible for them in the early days was performance, since it ran much faster than Python or Java.

However, as they needed to go deeper into horizontal scalability and performance, they realized that some of the most important pieces of web scalability – load balancers, messaging queues, etc are all based on top of Linux (for example, they use HAProxy for load balancing and Redis for queuing). As a result they built a lot of tools in-house to be able to interoperate with these sytems.

This is what Jeff has to say about sharing software in the .NET world.

You can certainly build open source software in .NET. And many do. But it never feels natural. It never feels right. Nobody accepts your patch to a core .NET class library no matter how hard you try. It always feels like you’re swimming upstream

I no longer live in the Microsoft universe any more. Right, wrong, good, evil, that’s just how it turned out for the project we wanted to build.

However, I’d also be lying if I didn’t mention that I truly believe the sort of project we are building in Discourse does represent most future software. If you squint your eyes a little, I think you can see a future not too far in the distance where .NET is a specialized niche outside the mainstream

Talent Trends

jobgraph2The graph from Indeed.com job search trends gives an indication of talent availability along PHP, C# and Java.. along with .NET and JVM languages like Scala and F#.

It is therefore a question of how you would acquire, retain and motivate your talent over the lifecycle of your engineering roadmap.

Homogeneity vs Interoperability

The main issue is not one of building the entire stack in Java (as opposed to Ruby or any of the hundreds of languages out there). The main issue is interoperability – because of the platform tie down of .NET, there are several hoops that need to be jumped to get an effective stack.

For example, one will invariably use HAProxy, Varnish, Memcached or Redis – all of them run on Linux and will add to the complexity of managing your overall stack (which started off primarily around Windows)

Impact on Service Oriented Architecture/Distributed Systems

  • Deployment & Management Systems - One of the biggest sources of complexity in a distributed systems framework is service orchestration. This is a problem both at production as well as during development (how do you setup a stack combining tens of services on a developer’s laptop). The Linux world has a plethora of systems which solve this problem with space age sophistication – Puppet, Chef, Ansible, etc. are all some of those. There are not many such tools in the Microsoft world.
  • Cloud Compatibility –  Except for Microsoft Azure (which is looking to be suffering from the same Silverlight risk mentioned above), all the other cloud platforms support stacks which are difficult to interoperate with .NET systems. For example, it is tricky to use Amazon RDS (even though it is currently built on SQL Server 2008 R2) from an ASP.NET deploy on EC2.
  • Self contained systems - License costs begin to impact as you begin to build a distributed system with self contained databases.

 

People have their opinions – especially when it comes to editors (Vim rules!) and platforms. There are some who are overly dogmatic about it . Yet, objectively thinking, I think I make a strong case to rearchitect away from .NET frameworks in a way that is easily justifiable.

How to think about web services APIs – REST and SOAP

Over the past couple of years, I have architected or consumed web services APIs inside various web applications. And during all that time, I have never really thought about a formal approach towards API design.

A lot of what I did was subconsiously picked up from the way Rails controllers were architected, or the way that Amazon APIs were built up. However, it stands to reason to analyze the fundamental philosophy behind that design.

I typically hear arguments like this – “REST is used for lightweight architectures, while SOAP should be used for complex architectures involving Transactions, Security, etc”

Duh.

Amazon – perhaps the backbone of the entire web services world implements a highly secure, authenticated+authorized web services framework on top of a REST-like API. You particular vendor might claim the above, but it isnt true in general.

Is it truly REST ?

HTTP is fundamentally two things – discoverable and stateless. You want something – you ask for it and http will either give it to you, or tell you what’s wrong. REST is very much like that.

SOAP – a philosophy predominant in the .NET world – is a philosophical first cousin to HTTP (and by extension REST). It basically takes both the essential elements of HTTP/REST – viz. the URL and the Header – and packages them as a XML document to be sent over a POST request to a single endpoint. So it kind of behaves like a single page web app – you dont know where you are with a single look.

Why is this even important ? There is a great presentation from MollyRocket, that leads you to the answer – reuse of components. Effectively, reuse is a function of time – a single endpoint with a complex set of instructions sent to each effectively nullifies the way you can architect for reuse.

Let’s understand what that means from an example of Object Oriented programming:

Transferring state vs transferring objects

In classic OO design philosophy – objects have state and function.

When you work with REST apis, you are just playing around with state – the functions are pre-defined: GET, POST, PUT, DELETE. Let’s say you have an object in your database – User. You now need someway to combine two users, so what do you do ? You define a resource /combineusers and possibly call POST with id=1 and id=2. Thinking in OO parlance, what you have just done is set the state of the CombineUsers class to id=1&id=2. This class, in turn, could have simply inherited/extended from the Users class. In fact, at this point, you can completely do away with traditional inheritance paradigms and instead bring in stuff like monkey patching to achieve the same result (IMHO – one of the reason, functional languages like Ruby work beautifully here).

How would you have done the same thing with SOAP ? Well you would have composed an XML like the following :

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
 <soap:Body>
  <gs:CombineUsers xmlns:gs="urn:CombineUsers">
   <id1>1</id1>
   <id2>2</id2>
  </gs:CombineUsers>
 </soap:Body>
</soap:Envelope>

But that is NOT idiomatic SOAP. What you really need to do (as per the “true way”) is request the entire Object on your client side and call seemingly native functions on both (which transparently call remote calls). Effectively, you would have done this:

$client = new SoapClient(null, array('location' => "http://localhost/soap.php",'uri'      => "http://test-uri/"));
$client->SomeFunction($a, $b, $c);
//or
$client->__soapCall("SomeFunction", array($a, $b, $c));

On the surface, it doesnt look too different. Now, think about how you would implement that on your remote side – every call to request an object has to send along with it (at a minimum), all applicable method calls. This becomes an even bigger clusterfuck, when you have to deduce “applicable method calls” on the basis of authorization and ACLs.

Suddenly, not only does your class become invariant, but also every one of your methods – because the whole world has access to them. How on earth are you supposed to refactor ?

I havent worked very deeply with SOAP server architectures, but I dont even think you can inherit classes here – you are always afraid of what methods you might inadvertently expose. So you have to start engineering in terms of interfaces , etc.

A list is an object.

With REST, you simply throw away the concept of working with objects and instead think in terms of state. The big advantage that comes here is that on the client side, you can model this state as a list.. or a map… or as a string. As long as you model the state correctly and transfer it to the server, nobody cares what the underlying object model is. Which is why REST based clients are easy to implement in the most quirkiest of programming languages.

The mystical compiler and interface definitions

One of the most popular arguments in favor of SOAP is type-checking and automatic datastructure creations by SOAP. Unlike most of the REST fanatics, I do agree on this part. In large enterprise-y infrastructures, this can be a godsend.

So is REST dead in the water in large teams ?

Not really. If you think about it, effectively what you want is what I like to call serialization compatibility.

Enter Thrift.

Thrift is a binary serialization protocol that is agnostic of the kind of web technologies you are using and the kind of underlying protocol that you are leveraging. It natively understands a few different basic types and generates interface wrappers for them (in a variety of languages – Ruby, Python, .NET, etc.).

Using Thrift over REST effectively nullifies the advantage that SOAP had, but gives you the advantage of type-checking. What’s more, REST allows you to specify the wire-format – so you can start off with JSON and move to Thrift once you are comfortable with your client.

The best part – Thrift is 600% faster than native XML/JSON.

Art of documentation

If you are part of an enterprise that is implementing a web-services rearchitecture, then the best way to answer “documentation???” questions is by two things – dogfooding and testcases.

Testcases: One of the problems with SOAP is discoverability. This is the endemic problem of .NET based architectures which are caught in a cycle of SDK based toolkits – loosely based on SOAP. If you want to use them, you need to write code in a supported SDK and instantiate the objects (remember what I wrote above ? you are getting the entire “object”).

Compare that to REST, where you can “GET” the state from your commandline

curl -XGET http://ifconfig.me/ip

What this effectively buys you is that you can get away very effectively by just documenting the list of end-points (“resources“). Your testcases should be able to give a hint of what is the expected output.

If you are using a interface language like Thrift, then the IDLs for the APIs could be useful, but they are effectively transparent to the client (since they only impact the wire).

Dogfooding: It effectively means eating your own food before feeding the dog. One of the best examples of enterprise rearchitecture and dogfooding was set by Jeff Bezos, who effectively threw a fit:

So one day Jeff Bezos issued a mandate. He's doing that all the time, of course, and people scramble like ants being pounded with a rubber mallet whenever it happens. But on one occasion -- back around 2002 I think, plus or minus a year -- he issued a mandate that was so out there, so huge and eye-bulgingly ponderous, that it made all of his other mandates look like unsolicited peer bonuses.

His Big Mandate went something along these lines:

1) All teams will henceforth expose their data and functionality through service interfaces.

2) Teams must communicate with each other through these interfaces.

3) There will be no other form of interprocess communication allowed: no direct linking, no direct reads of another team's data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network.

4) It doesn't matter what technology they use. HTTP, Corba, Pubsub, custom protocols -- doesn't matter. Bezos doesn't care.

5) All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.

6) Anyone who doesn't do this will be fired.

7) Thank you; have a nice day!
- from a now deleted blog post by Steve Yegge

if you eat your own dogfood, I dont see why there will ever be a documentation problem (there still will be :twisted: , but much less than otherwise)

Is this perfect ?

No, it is not. But I would say that a perfect solution is marketing sell from an enterprise vendor. What you should be looking for is understanding the problems and defining your organization to work with it, instead of shooting for the stars and having it fall on top of your head.