Proxy or no proxy for your API? That’s not the question. One of the common debates when setting up an API management system is whether to proxy your traffic in some way before it reaches your application stack – and there are pros and cons to different architectures. however that’s not the goal of this post.

If a proxy based architecture fits your requirements the next question remaining is what proxy should we put in place? Yes, there are options, in fact there are many. On one hand there are the closed proprietary proxies offered by different vendors. On the other hand there are open-source proxies like Varnish, Squid, Apache, Nginx, etc.

Proxies are basic components of today’s technological stack, just like web servers or load balancers. At 3scale we believe that is better to integrate, extend and support open source solutions rather than reinventing the wheel and charge for it. Is your API using a proprietary web server? Probably not. Apache and Nginx alone have more than 75% market share as of 2012. Why proxies should be different?

Open source is not a matter of licence costs, let alone a matter of being cool. Is a matter of production ready software, proven high-performance, reliability, community support, plenty of documentation and examples and public scrutiny to detect and fix security vulnerabilities, etc.

So, which open source proxy should you look at? There is no right or wrong answer here. It depends on your current stack, on the expertise on a particular framework, it even depends on personal taste. What is important is that there are terrific solution readily available.  Squid is one, the proxy modules for both Apache and Nginx are also very interesting specially if you are already using Apache or Nginx for your web server. But perhaps the most famous one is Varnish, never heard of it? Time to get updated.

Varnish is part of the infrastructure at power-houses such as Twitter, Facebook, Heroku, LinkedIn.  But proxy/caching is not a used widely, also beyond the biggest sites;  Softonic, Peoplebrowsr, Magento among many others also use it to power-up their backend.

According to Varnish’s web page,

Varnish is a web application accelerator. You install it in front of your web application and it will speed it up significantly.

This one-liner on their main page does not make justice on how powerful Varnish is. Architecturally Varnish is a proxy that stays in front on your HTTP stack (roughly speaking, the tech stack that ends up serving web pages and/or API requests).

Varnish usually is placed in front of a web server like Apache, Nginx, Tomcat, IIS, Thin, etc. Whatever web server you use, you can stick a Varnish in front of it like that

After the initial setup, which takes minutes to have up and running, Varnish will cache the heck out of your application, whether it is web pages or API calls. The caching layer is extremely configurable and can be used to serve both write or read heavy workloads. The must read blog High Scalability has a recent entry on Varnish that says it all, and better.

Can we use varnish to scale our API? Absolutely, in fact, you should. API requests are web content and as long as you don’t have a write only API the benefits of an open, well maintained and secure proxy + caching layer will save you on the load of your servers at the expense of less revenue for Amazon/Rackspace/Softlayer.

Our API does not have scalability problems as of today, we are still low on traffic. True, you don’t have to use it, but serving a request from the varnish cache can be order of magnitude faster than if the API request has to go all the way down to your application code. Latency matters, and you don’t want your API to be “slowish”.

So if your API needs a proxy, can you use 3scale? Absolutely, if your choice of proxy is Varnish you only have to add the new 3scale module for it, available on github. With the 3scale module for Varnish basically we are adding the 3scale management and control layer to our API at the level of the proxy.

Once the 3scale varnish module is installed Varnish will check against 3scale for the validity of the API request. If positive, Varnish will serve the result from its cache or fetch the data from your origin server, finally Varnish will report to 3scale to maintain the consistent state and real-time analytics. Needless to say that the connection between your local Varnish proxy and 3scale’s backend can be done asynchronously so that latency is kept at a minimum, again thanks to Varnish’s caching abilities.

In follow up posts we will give a step by step guide on how to integrate your API with Varnish. Also, we will dig further on the powerful VCL (Varnish configuration language) and its third-party modules support. Besides making it extremely easy to integrate your API with external services like 3scale. Varnish can also be helpful in domain specific tasks like basic transformation and security.

Stay tuned.

Some of our favorite apps were:

• SiftOSC: which used Sift RoboCubes to control sounds in wierd and wonderful ways.
• Paul v’s BillBoard: which pits EchoNest Meta Data against Billboard Magazine on Grammie predictions – go Paul!
• Audible Harvest: The demo timed out the idea was cool – pulling
• SongRoulette: Chatroulette for songs – powered by Spotify
• KloutRock: a spotify app which helps you follow your most influencial music heros.
• SideTrack: a cool mashup of input, lyrics and randomness to pop new songs.
• SeatTrip: integrating Seevl, Email and Seatwave for tickets as you travel.

Congrats to Sidetrack won the 3scale GoPro Prize!! Props to all the organisers and the hosts TokBox – a great couple of days!

• Musixmatch: great lyrics API which allows you to get legal lyrics into your app and grab great data like lyrics annotations for videos – over 1 Billion API calls in 2011. 
• Seevl: power up music discovery with seevl for matchmaking by music similarity and genre. The API lets you embedseevl recommendations into any kind of application. 
• Peachnote: the newest of the three – Peachnote which allows dynamic search for melodies in music corpuses from the Petrucci Music Library, the Library of Congress and the Duke University Library.

There’s a GoPro HD2 Camera up for grabs for the best app which uses any of these APIs – so you’ll have some cool imagery to combine with your SF Hack Day tunes!

Many companies have amazing data assets and these form the core value of their business for their customers. APIs provide the means to create new distribution channels for this data. Specifically, APIs potentially allow distribution to be turned on its head – rather than pushing data out to customers and their bespoke interfaces, APIs enable customers to pull in the data they need and to integrate it into the interfaces where they need it. The real value behind this includes:

• Enableing new types of customers to access the data who are unable to use the existing interfaces.
• Enableing customers to specialize their interfaces to the data – tying them more tightly to the data stream and less likely to switch to other providers.
• Enable finer grained access to data sources for customers and enable new types of service/data bundling – and potentially enable new business models.
• Enabling new potential resale channels for the data – which may be aggregators or multi-data source mobile applications.

As the interfaces to data change increasingly quickly on the web and mobile applications, having an API strategy in place provides a stable platform for new products both from within the company and from outside.

There seemed to be concern on the panel about loss of control of the data or loss of brand visibility. These are legitimate concerns, but ultimately unfounded – since there is no reason why a data company should not put significant restrictions on data usage, branding and resale. The company remains in complete control of the data and can set the rules for usage. In many ways, access to data via API rather than the web makes it easier to control who is accessing which data v’s web based access. Many APIs also allow access to only highly trusted parties or specific customers in restricted ways – far from the extreme of completely open APIs. Restrictions on data access can be enforced either by in-house code or via infrastructure solutions such as 3scale or Mashery.

Another concern raised was that the data provision companies such as Bloomberg also often already do an excellent job of providing data terminals and user interfaces to their customers – and that there is no need to outsource. There is no doubt that successful company APIs often also include powerful standard clients/interfaces – and these interfaces are often important differentiators themselves. These interfaces will continue to be critical, however it is rare that companies are in a position to fulfill every integration need their customers may have, every interface need and be able to reach all the audiences they wish to reach.

The API therefore provides the platform for extensions in areas which may not be high enough value yet to tackle but which add value to customers and may one day become larger opportunities. Such extensions also help create a powerful ecosystem around the data on offer and enable customers to put down deeper roots with the data company.

Obviously an API strategy requires thought about business consideration as well as technical –  but as a platform APIs are likely to add very significant value to content and data distribution.

You can download the Swagger framework from http://swagger.wordnik.com/.

Swagger Framework Overview

Swagger is language-agnostic. However Wordnik (the creator of the Interactive API documentation) has released Swagger implementations in Scala, Java, and HTML5 and client generators are currently available for Scala, Java, Javascript, Ruby, PHP, and Actionscript 3. Wordnik says that more client support is underway.

• Automated, consistent generation of clean, beautiful, interactive API documentation
• Generation of client code and SDK in multiple languages
• Feeding into an industry wide API discovery language that both developers and non-developers can use

If you would like to know how Swagger can be used with 3scale’s API Management Platform contact us at info@3scale.net

Here is a full list of the demo apps:

• Movable Feast: for making guacamole on stage in real time, with realtime energy use notifications.
• Audit Trigger: quick feedback on whether your building might need an energy audit – built on energyscoregards.
• CleanGPA: trying to conquer energy apathy through healthy competition: http://cleangpa.com/.
• NYC Municipal Building Face-Off: 2 million sq feet of building space – visualising energy efficient and inefficient buildings  http://nycbldgs.com/.
• WattQuiz: the energy equivalent of http://www.freerice.com for energy markets in order to educate user on energy related questions using the Genability API.
• Automatic Wireless Thermostat: using android geo-fencing to control your home thermostat.
• Econify; comparison shopping for energy efficiency, including the calculation of the payback timeline per appliances.
• Watt: assigning a number to every individual and zip codes.
• GreenCarrot: help improve energy awareness using social techniques based on tendril and facebook, including notifications and to-do tasks, http://www.greencarrot.us/.
• 1v99: highlights the differences in income and consumption between the wealthiest 1% and the 99%.
• Mosaic: trying to make solar social - visualizing solar energy conversations and crowdfunding for solar projects.
• Solar List: provides price calculation for solar intallations.
• SoBi – TripWatcher: weightwatchers for transportation.
• Tendril Remote Control: switching in house appliances based on distance from home.
• eMotivator: comparative data for energy usage between friends.

Congrats to all the apps. Ideas create hacks, hacks create solution – roll on clean tech hacks!

• better control over Email Workflow with the new ability to customize all message content
• it is easy to obtain PCI compliance if you charge for your API, because security for handling credit card data has been boosted.

We hope you enjoy exploring the new features and we would love to hear your feedback at support@3scale.net.

Email Templates

Every single email notification to both your developers as well as your admin users can now be customized to suit your unique workflow and desires. The Emails tab lists all potential emails, but many of them may not be used, depending on how your workflow is configured:

Change to Credit Card handling

If you use the automated 3scale billing module, you will soon see a change in the way in which credit card data is handled. In future all requests for users to submit credit card data will be redirected and handled directly by the interface belonging to your chosen payment gateway partner.

We’re excited that we can talk about one of the newest innovations we’ve been cooking up here at 3scale: API delivery over professional global CDNs powered by Akamai and CDNetworks. This means that with 3scale you can now:

• Connect your API to a professional grade CDN at blazing fast speeds.
• Serve content into the CDN’s caches using standard caching controls.
• Have your customers access that content directly from CDN edge nodes without the need for calls back to your origin servers.
• Manage your API Channel with tools similar to those you’re already using for your Web Site.

This speeds up delivery to your users, offloads traffic from your origin servers for scalability. Both GET and POST requests can be controlled within the CDN and 3scale’s powerful management tools allow you to keep full control of API Access, keys, rate limits, analytics across the whole delivery platform.

How it works: methods calls to the API can be cached within the CDN using standard caching controls on a push or pull basis. Keys for API access with detailed usage policies can be controlled from within the 3scale control system as well as auto-generated via the API Developer portal. Apps access to the API via the CDN edge nodes that make calls directly to the 3scale control nodes in order to enforce policies without needing to communicate with the origin.