The value of having a platform for stateless applications that ties together with a platform for stateful services is greater than the sum of its parts. Utilizing the Cloud Native Apps nature of Cloud Foundry to create new, innovative products while making sure your customer data and internal company information lives in a stable, reliable and trusted environment is of utmost importance. In this blog post we’d like to show you how to merge these concepts together.
If you’re familiar with Cloud Foundry, then you probably know it’s focused on running stateless applications – hundreds or even thousands of them. These applications can be scaled up or down to handle a myriad of different workloads, all while storing data outside the application. This makes the applications adhere to the 12-factor manifest where configuration and data should always be kept outside where the application runs.
When talking with customers we’ve noticed a common theme around users gravitating towards a common platform for new applications, and that platform is Mesos. Mesos is also heavily focused on stateless applications but thanks to work we’ve done with REX-Ray and mesos-module-dvdi we’ve seen an increasing interest in running stateful services as well.
Can we somehow merge these concepts together? Running stateful services in Mesos and stateless applications in Cloud Foundry, with some awesomeness sprinkled in?
Of course we can
First, let’s get Cloud Foundry up and running. There’s a great micro-version of Cloud Foundry that can be found here. Download it, unzip the file and start Cloud Foundry by following the instructions in the repo.
Once you have Cloud Foundry deployed and you can push stateless apps to it, you’ll want to add in databases and other stateful services to the mix. We’ll host those stateful services on Mesos. To do this, head over to our vagrant repo – we’ll be using the playa-mesos setup in there.
In the config.json file for playa-mesos there’s a setting to enable Consul, make sure that’s set to
true on all nodes. We’ll get back to that in a bit. Next, start your Mesos environment by running
You should now have Cloud Foundry, Mesos and Marathon running in your local environment, awesome!
Next, it’s time to add a stateful service. We’ll use the key-value store Redis for this. For proper persistence we of course have REX-Ray and Mesos-module-DVDI module already installed and enabled in the Mesos environment, so we should be all set. If perhaps you do not quite yet, you can download these here: REX-Ray and mesos-module-DVDI.
To launch Redis with proper datastore backing you can use this JSON manifest:
Launch Redis using the following command:
curl -X POST http://10.141.141.10:8080/v2/apps
-d @redis.json -H "Content-type: application/json"
By looking at the Marathon interface over at http://10.141.141.10:8080/ui/#/apps you should see Redis being deployed.
Connect them together!
Now let’s see how we can connect our stateless apps to this redis service.
Should we do this manually by adding IPs and ports to the code? Absolutely not
Instead, we’ll use Consul – a great piece of software that enables automatic service discovery of everything that’s in your data center. Cloud Foundry uses Consul internally to keep track of everything it runs, and here we use another instance of Consul to keep track of everything we’re running through Marathon.
When you started up the Mesos cluster you actually also created a Consul cluster (thanks to the
true setting above) and another piece of software called marathon-consul that ties Marathon and Consul together. Everything you run through Marathon with the tag “consul” gets registered in Consul, you can see an example of how that’s done in the redis.json above.
Now, let’s have a look at what happens here:
- You push a Redis manifest up to Marathon
- Marathon reads the manifest and asks Mesos to run the application
- Mesos sees the resource requirements (including storage), and allocates them using posix/cgroups, DVDI and REX-Ray
- When the application is running, Marathon reports back with a green status light
- marathon-consul now adds information about Redis into Consul
The information about Redis stored in Consul can be pulled by running the following:
$ http GET 10.141.141.10:8500/v1/catalog/service/redis
HTTP/1.1 200 OK
Date: Fri, 22 Apr 2016 19:53:19 GMT
You can see we have an IP address and a port registered for the Redis service. Now how can we use that with our application?
First, we need to make sure that the apps on Cloud Foundry are allowed and able to talk to the services on Mesos. To do that we update the security-group and the dnsmasq service. In the pcfdev folder, run the following commands:
$ wget -nc -nv https://gist.githubusercontent.com/jonasrosland/c70cce115ddcb2422ee5f71800aed1d5/raw -O public_networks.json
$ cf update-security-group public_networks public_networks.json
Updating security group public_networks as admin
$ vagrant ssh -c "sudo tee /etc/dnsmasq.d/10-consul >/dev/null << EOF
$ vagrant ssh -c "sudo service dnsmasq restart"
Now your Cloud Foundry applications are allowed and ready to connect to Mesos services. Let’s take our example application for a spin!
The application we’re using can be found here: simplehttp. Let’s have a look at it:
from flask import Flask, render_template
app = Flask(__name__)
CONSUL_HOST = os.getenv("CONSUL_HOST", "no.host")
CONSUL_PORT = os.getenv("CONSUL_PORT", 0)
CONSUL_DC = os.getenv("CONSUL_DC", "nodc")
consul = consulate.Consul(host=os.environ['CONSUL_HOST'], port=os.environ['CONSUL_PORT'], datacenter=os.environ['CONSUL_DC'])
data = consul.catalog.service('redis')
json_str = json.dumps(data)
resp = json.loads(json_str)
redis_address = (resp['Address'])
redis_port = (resp['ServicePort'])
r_server = redis.Redis(redis_address, redis_port) #this line creates a new Redis object and
counter = r_server.get('counter')
return render_template("index.html", CONSUL_HOST=CONSUL_HOST, CONSUL_PORT=CONSUL_PORT, CONSUL_DC=CONSUL_DC, redis_address=redis_address, redis_port=redis_port, counter=counter)
app.logger.error('Server Error: %s', (error))
if __name__ == "__main__":
app.run(debug=False,host='0.0.0.0', port=int(os.getenv('PORT', '5000')))
The application flow looks like this:
- Queries the local DNS in Cloud Foundry to look up the Consul environment
- Sends a query to Consul to gather information about the Redis service
- Connects to Redis, adds an incremental value to the key “counter”
- Retrieves the key and publishes the content on the webpage
You might see that the app itself actually has querying functionality built into the function we’re calling. This is a quick and dirty way to enable resiliency to the application since it will automatically pick up new Redis information if the database has to move, restart or has a failure and recovers from it somewhere else.
Make sure to
git clone the entire application, then all you need to do is a
cf push to push it to the Cloud Foundry environment. It will look something like this:
$ cf push
Using manifest file /Users/jonas/Developer/jonasrosland/demos/simplehttp/manifest.yml
Creating app simple in org pcfdev-org / space pcfdev-space as admin...
Creating route simple.local.pcfdev.io...
Binding simple.local.pcfdev.io to simple...
Uploading app files from: /Users/jonas/Developer/jonasrosland/demos/simplehttp
Uploading 18K, 10 files
Starting app simple in org pcfdev-org / space pcfdev-space as admin...
Downloaded python_buildpack (254M)
Successfully created container
Downloading app package...
Downloaded app package (5.9K)
1 of 1 instances running
App simple was started using this command `python simple.py`
Showing health and status for app simple in org pcfdev-org / space pcfdev-space as admin...
requested state: started
usage: 256M x 1 instances
last uploaded: Fri Apr 22 21:05:58 UTC 2016
state since cpu memory disk details
#0 running 2016-04-22 05:06:55 PM 0.0% 0 of 256M 0 of 512M
You can now go to simple.local.pcfdev.io and you should see the following screen:
As you can see, the application automatically finds its database, and when you refresh the page you’ll see the Redis counter go up, pretty cool!
Now for some resiliency testing, let’s restart Redis. To do that, head to the Marathon interface at http://10.141.141.10:8080/ui/#/apps/%2Fredis and click “Restart”.
It might end up on the same host but on a different port, or a completely different host. Up to you to experiment.
After restarting, you can refresh the simplehttp page and see both the new connection information for Redis and the value increasing!
Here’s what we just did together…
- Ran a stateful service (Redis) on Mesos
- The stateful service had proper storage support thanks to REX-Ray and mesos-dvdi-module
- Restarted the stateful service results in the data moving with the service
- The connection information for Redis was automatically stored in Consul
- Launched a stateless application on Cloud Foundry with no information on where the stateful service was located
- The application queried Consul for connection information
- Once the application had the information, it connected to Redis
- The application is ready to store and retrieve data
If you liked this demo, please check out the repos that made it possible: