
Sign up to save your podcasts
Or


Building data science teams
Data science teams need people with the skills and curiosity to ask the big questions.
http://radar.oreilly.com/2011/09/building-data-science-teams.html
People You May Know (PYMK) LInkedin, Facebook
Netflix and Zynga
Google, Amazon,
A recent report from the McKinsey Global Institute says that by 2018 the U.S. could face a shortage of up to 190,000 workers with analytical skills.
http://tech.fortune.cnn.com/2011/09/06/data-scientist-the-hot-new-gig-in-tech/
http://www.dataspora.com/2011/09/data-scientists-or-data-composers/
-------------------------------------------------------------------------
New CycleCloud HPC Cluster Is a Triple Threat: 30000 cores, $1279/Hour, & Grill monitoring GUI for Chef
http://blog.cyclecomputing.com/2011/09/new-cyclecloud-cluster-is-a-triple-threat-30000-cores-massive-spot-instances-grill-chef-monitoring-g.html
We have now launched a cluster 3 times the size of Tanuki, or 30,000 cores, which cost $1279/hour to operate for a Top 5 Pharma. It performed genuine scientific work -- in this case molecular modeling -- and a ton of it. The complexity of this environment did not necessarily scale linearly with the cores.
c1.xlarge instances 3,809
cores 30,472
RAM 26.7-TB
AWS Regions 3 ( us-east, us-west, eu-west )
Compute Years of Work 10.9 years
Spot Instances at an average cost of 0.286 USD / instance / hour (0.036 USD / core / hour). Compare that to the 0.68 USD / instance / hour for the same On Demand instance. That’s 57% savings!
-------------------------------------------------------------------------
What Exactly is Complex Event Processing Today?
http://blog.cloudeventprocessing.com/2011/09/18/what-exactly-is-complex-event-processing-today/
Colin Clark...
-------------------------------------------------------------------------
https://github.com/nathanmarz/storm/wiki/Rationale
Storm is a distributed realtime computation system. Similar to how Hadoop provides a set of general primitives for doing batch processing, Storm provides a set of general primitives for doing realtime computation. Storm is simple, can be used with any programming language, and is a lot of fun to use!
The lack of a "Hadoop of realtime" has become the biggest hole in the data processing ecosystem.
-------------------------------------------------------------------------
Building a Devops team
http://agilesysadmin.net/building-a-devops-team
Brian Henerey, from Sony Computer Entertainment Europe.
First interview - remote technical test
Instructions...
Ec2 instance .. install Wordpress with a broken Mysql install
Tomcat log scraping...
Using screen to watch them...
Round 2 - Face to face interview
Whiteboard test
Pair programming
-------------------------------------------------------------------------
How to Think Like a Computer Scientist
http://greenteapress.com/thinkpython/thinkCSpy/
Learning with Python
-------------------------------------------------------------------------
Node.js and MongoDB on Ubuntu
http://cloud.ubuntu.com/2011/09/node-js-and-mongodb-on-ubuntu/
haproxy to catch inbound web traffic and route it to our node.js app cluster
mongodb for app storage
With a sample application....
-------------------------------------------------------------------------
First steps with Cloud Foundry on Amazon EC2
http://www.cloudsoftcorp.com/blogs/first-steps-with-cloud-foundry-on-amazon-ec2
Setting up an IP address and domain name
Setting up an IP address and domain name
Making it start the right modules at boot
NoSQL Benchmark
http://blog.cubrid.org/dev-platform/nosql-benchmarking/
Yahoo Cloud Servicing Benchmark
Basic operations are Insert, Update, Read, and Scan. There are basic workload sets that combine the basic operations, but new additional workloads can also be created.
This article contains tests conducted on the following products and versions.
Cassandra-0.7.4
Although Cassandra’s latest version is 0.8.0, we have decided to use the previous version known to be stable. Because when testing with the 0.8 version, the gossip protocol between nodes malfunctioned and the node up/down information was incorrect.
HBase-0.90.2 (Hadoop-0.20-append) The HBase-0.90.2 (Hadoop-0.20-append) was selected because, if not the Hadoop-append version, there may be problems on decreased durability in HDFS.
MongoDB-1.8.1
Insert, Read Only and Read and Update
Insert - Cassandra kills
Read and update Cassandra beats HBase by a little
Read Hbase wins of course but only by a little against Cassandra
Mongo get blow out...
Which leads me into .. why I would love to make this event...
------------------------------------------------------
Using Cassandra, Brisk, and Mahout to Manage Time Series, and Predict Future Events
http://predictingfuture.eventbrite.com/
Datastax ... Brisk a cassandra based Hadoop...
------------------------------------------------------
What is glu?
http://linkedin.github.com/glu/docs/latest/html/index.html
glu is a free/open source deployment and monitoring automation platform.
a glu agent is running on each of those nodes
ZooKeeper is used to maintain the live state as reported by the glu agents (blue arrows)
the glu orchestration engine is the heart of the system
Glu Script is a Groovy Class with named closures for the actions... (can be groovy or java)
install, configure, start, stop, unconfigure and uninstall
The doc is pretty cool .. however, when I started getting into the state machine stuff I had to stop...
Orchestration .. Zookeeper to build live state, compare live and desired state.
generate delta
------------------------------------------------------
NODE.JS AND THE JAVASCRIPT AGE
http://metamarketsgroup.com/blog/node-js-and-the-javascript-age/
Three months ago, we decided to tear down the framework we were using for our dashboard, Python’s Django, and rebuild it entirely in server-side JavaScript, using node.js. (If there is ever a time in a start-ups life to remodel parts of your infrastructure, it’s early on, when your range of motion is highest.)
This decision was driven by a realization: the LAMP stack is dead.
1991-1999: The HTML Age.
2000-2009: The LAMP Age.
2010-??: The JavaScript Age.
------------------------------------------------------
From $0-100million with no sales people. The Atlassian 10 commandments for startups.
http://blog.businessofsoftware.org/2011/09/from-0-100million-with-no-sales-people-the-atlassian-10-commandments-for-startups.html
Jira, Confluence
3 ppl to 300 ppl...
Start with two founders.. 50/50
Bootstrapping .. first round is 60M
-Sell itself, affordable, global, open
-Use your own product.... Passionately use your own product...
-Measure everything... Capture everything.... even if you can’t analyze
-Test everything... 5 users free .. raised money for charity
-ABM... ... always sponsor the beer at conference.. like Dyninc...
-Send stuff in the mail.. t-shirts...
-Make everything into a campaign.. Turned hiring into a marketing campaign - .. send only 4 resumes otherwise you are black listed...
-Don’t be afraid to let your first product will fail..
------------------------------------------------------
Devops Dude of the Week....
Jordon Sissel
FPM and Logstash and now...
eventmachine-tail
https://github.com/jordansissel/eventmachine-tail
Jordon Sissel..
This project contains two EventMachine extensions.
First, it adds an event-driven file-following similar to the unix ‘tail -f’ command. For example, you could use it to follow /var/log/messages the same way tail -f would.
Second, it adds event-driven file patterns allowing you to watch a given file pattern for new or removed files. For example, you could watch /var/log/*.log for new/deleted files.
For logstash, the log agents were event-driven using EventMachine. The log agents mainly get their data from logfiles. To that end, we needed a way to treat log files as a stream.
There’s a ruby gem ‘file-tail’ that implements tailing, but not in an event-driven way. This makes it hard to use in EventMachine programs like logstash.
Thus, eventmachine-tail was born.
Further, the usage patterns for logstash required the ability to watch a directory (or a file pattern) for new log files.
rtail -x "*.gz" "/var/log/**/*"
-
$3m Wellington rail project behind schedule
http://www.stuff.co.nz/dominion-post/news/5606019/3m-Wellington-rail-project-behind-schedule
KiwiRail said Project Sirius, a $3 million project to install an IBM asset management system, is six months behind schedule.
------------------------------------------------------
LexisNexis Releases Code for Its Hadoop-Killer
http://www.devx.com/DailyNews/Article/47281?trk=DXRSS_LATEST
LexisNexis Risk Solutions' division HPCC Systems has announced that it is open sourcing the code for its High Performance Computing Cluster (HPCC) software. HPCC is a data-processing-and-delivery solution that the company is marketing as an alternative to Hadoop. HPCC includes two major components: Thor, which analyzes large datasets in a manner similar to Hadoop, and Roxie, which is closer to a traditional RBDMS or a data warehouse.
------------------------------------------------------
The Apache Software Foundation Announces Apache Whirr as a Top-Level Project
https://blogs.apache.org/foundation/entry/the_apache_software_foundation_announces15
Apache Whirr provides a Cloud-neutral way to run a properly-configured system quickly through libraries, common service API, smart defaults, and command line tool. Whirr is being used for proof of concepts and a way to try out new Cloud services utilizing a variety of Apache products that include Hadoop, HBase, Cassandra, and ZooKeeper. An example of this is enterprise software providers Cloudera, who use Whirr to make it easy to try out their CDH product and run distributed clustered services.
------------------------------------------------------
Chef Hack Day - Seattle
Saturday, September 24, 2011 from 9:00 AM to 5:00 PM (PT)
Seattle, WA
------------------------------------------------------
Joyent arms cloud for death match with Amazon
http://www.theregister.co.uk/2011/09/15/the_new_joyent_cloud/
The pixar of cloud Jason Hoffman: Chief Scientist, Founder - PhD in Molecular Pathology
Mark Mayo: CTO...
A month after open-sourcing what it calls "the first major hypervisor" to arrive in half a decade, cloud computing pioneer Joyent has added this hypervisor to its flagship service, allowing Linux and Windows applications onto the Joyent Cloud for the first time.
Joyent and its firebrand CTO told the world they had ported the KVM hypervisor from Linux to SmartOS. They promptly open-sourced the code in an effort to "make the world a better place", and now they've rolled the hypervisor into a new incarnation of the Joyent Cloud
The company claims that its SmartOS virtual machines are up to 14 times faster than comparable Amazon server instances
------------------------------------------------------
Rundeck And Nagios Nrpe Checks
http://www.dzone.com/links/r/rundeck_and_nagios_nrpe_checks.html
I’ve played with a few different jobs so far, including triggering Puppet runs across machines triggered by a Jenkins plugin. I’ve also been looking at running all my monitoring tasks at the click of a button (or again as part of a smoke test triggered by Jenkins) and I thought that might make a nice simple example.
My checks are written as Nagios plugins, and run periodically by Nagios. I also trigger them manually, using Dean’s NRPE runner script.
VMware vsphere provider to Fog
https://github.com/geemus/fog/pull/505
Libvirt integration for fog
https://github.com/geemus/fog/pull/484
http://jedi.be/blog/2011/09/13/libvirt-fog-provider/
added cloudstack support
https://github.com/geemus/fog/pull/456
Full AWS support sqs, sns, rds, elb, dns,cloudformation, cloud_watch
----------------------------------
Patrick is going crazy over there..
This is some crazy shit essay..
Automated Vmware ESX Installation - Bonus in Vmware Fusion
http://www.jedi.be/blog/2010/12/09/automated-vmware-esx-installation-even-in-vmware-fusion/
Using kickstart
After this exercise you should be able to completely script the installation of a Vmware ESX virtual machine and make it run inside Vmware Fusion.
-------------------------------------
Murder: Fast datacenter code deploys using BitTorrent
http://engineering.twitter.com/2010/07/murder-fast-datacenter-code-deploys.html
twitter eng.. .BitTorrent... Murder a 40 minute deploy.. 12 seconds! - in ruby and python also a great video on the blog.
-------------------------------------
Ruby for Jenkins Goes Pre-Alpha
http://blog.thefrontside.net/2011/09/13/ruby-for-jenkins-goes-pre-alpha/
The prject was started to make Jenkins fit the ruby comminity stlye...
not forced into using jruby, or maven.
boot a plugin written in pure Ruby into a Jenkins server w/o java or java knowledge.
-----------------------------------
Installing on RHEL/CentOS 5
http://support.cloudfoundry.com/entries/20237758-installing-on-rhel-centos-5
Decomposed the script based install and refactored it to work with centos and layed out the steps to use yum
4 ways to install
-----------------------------------
LexisNexis Releases Code for Its Hadoop-Killer
http://www.devx.com/DailyNews/Article/47281?trk=DXRSS_LATEST
HPCC Systems a division of lexisnexis risk solutions division.
open sourcing the code for its High Performance Computing Cluster (HPCC)
an alternative to Hadoop.
Thor, which analyzes large datasets in a manner similar to Hadoop,
Roxie, which is closer to a traditional RBDMS or a data warehouse.
-----------------------------------
Ensemble gets some Juju!
juju.ubuntu.com
We figured it should represent the complexities and mystery that often surround those skilled in the DevOps field, and be something that played on the same “u” sound and etymology as Ubuntu. Thus, “Juju” was born!
Devops Hackday with Cloudfoundry at VMWare about 300 ppl showed up.
First ting we did was everyone got the CF Micro on USB stick (cloud on a stick)
http://kb.dtosolutions.com/wiki/Cloud_Foundry_Micro
two teams about 9 guys did a puppet git to jenkins to cf automation pipleline.
Second team was git to jenkins to tomcat with zenoss
http://kb.dtosolutions.com/wiki/DTO_Hack_Day_09/08/2011
Takes aways for VMware was that it’s not just about the cloud. They thought
----------------------------------------------------------------
Podcast with Chris Pinkham, CEO of Nimbula, during VMworld 2011
http://neovise.com/podcast-vmworld-2011-nimbula-chris-pinkham-cloud-interview
Chris was formerly Vice President, IT Infrastructure at Amazon.com
Nimbula Director
Like I said that area is a blood bath. I know this when I was at Canonical tring to slep UEC and that was before HP, Dell, Citrix were in the game...
all the start ups...
10gen raises $20M for MongoDB in maturing NoSQL space
http://gigaom.com/cloud/10gen-raises-20m-more-for-mongodb-in/
has raised $20 million in a Series D funding round. Sequoia Capital Total venture backing to more than $31 million.
The post points out the Couchbase has raised around 30m as well. Damien Katz is the createor of Couchdb... was an orginally a Lotus Notes developer, then started the CB project, then IBM hired him to work on CB and now he is part of Couchbase...
http://www.infoq.com/presentations/katz-couchdb-and-me
------------------
$3m Wellington rail project behind schedule
a $3 million project to install an IBM asset management system, is six months behind schedule.
http://www.stuff.co.nz/dominion-post/news/5606019/3m-Wellington-rail-project-behind-schedule
When will this madness stop... I think about flightcast... Built an airline flight delay predictor that is in the high 80% tiles beased out of y combinator with three guys total capital probally less that 1/2 million...
No excuse for this kind of behavior...
-----------------------
dnsxd is an Erlang DNS server with a focus on DNS Service Discovery.
https://github.com/andrewtj/dnsxd
dnsxd's default datastore module is an interface to CouchDB
==============
Dustin Kirkland has two posts on installing Cloudfoundry on Ubuntu
sudo apt-get install cloudfoundry-server
http://blog.dustinkirkland.com/2011/08/howto-install-cloudfoundry-server-paas.html---
-----------------------
Devops Book List
http://devopsbooks.com/
The Visible Ops Handbook: Starting ITIL in 4 Practical Steps
Gene Kim
Release It!: Design and Deploy Production-Ready Software (Pragmatic Programmers)
Micheal Nygaurd
Cloud Application Architectures: Building Applications and Infrastructure in the Cloud (Theory in Practice (O'Reilly))
George Reese of Entratus
Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Addison-Wesley Signature Series (Fowler))
Jez Humble
Web Operations: Keeping the Data On Time
John Alspaw
Pulling Strings with Puppet: Configuration Management Made Easy
James Turnbull
Not on the list...
The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses by Eric Ries
Test-Driven Infrastructure with Chef by Stephen Nelson-Smith
The Last Lean Startup Bundle: 48 hours to claim $3,000,000 in prizes
http://www.startuplessonslearned.com/2011/09/last-lean-startup-bundle-48-hours-to.html
One book 750 worth of stuff. the 5 and 10 books kinda not so good...
HP Launches Private Beta Of Cloud Compute, Storage Services
http://www.crn.com/news/cloud/231600974/hp-launches-private-beta-of-cloud-compute-storage-services.htm
started a twitter discussion. I pointed out that I am pretty sure (certain) that they did not use opsware as a the deployment tool of choice...
meta point is that second gen CM tools are dead.. Tivoli, Bladelogic, Opsware... #puppet and #chef rule the cloud... no turning back...
Fault Tolerance and Protection
http://www.kitchensoap.com/2011/09/08/fault-tolerance-and-protection/comment-page-1/#comment-8339
GN&C FAULT PROTECTION FUNDAMENTALS
https://pub-lib.jpl.nasa.gov/docushare/dsweb/Get/Document-316/08-031+GN%26C+Fault+Protection+Fundamentals.pdf
Thinking outside the box...
-----------------------
Autometrics: Self-service metrics collection
http://engineering.linkedin.com/52/autometrics-self-service-metrics-collection
lnked in system
uses zookeeper for coordination.
Kafka
Amazon deploys every 11.6 seconds
Jon Jenkins
Amazon.com
June 16, 2011
http://www.youtube.com/watch?v=dxk8b9rSKOo
http://news.ycombinator.com/item?id=2971521
some gems in here.
nov 10 2010 all amazn web servers went ec2...
Makes a great argument for utility computing (i.. cloud)
Apllo deployment system.
one moth stats 11.6 depolys per second, peak 1079 in one hr,
10k avg sim deploys
30k peak.
But even their tooling reflects decoupling. Every tool follows the self-service model ("YOU do what you WANT to do with YOUR stuff"). Their deployment system (named Apollo, mentioned in the slides) and their build system, and their many other tooling, all reflect this model.
Cons. What happens is that you might be reinventing the wheel at Amazon. Often. Code reuse is very low across teams. So there's no shared cost of ownership at Amazon, more often than not. It's the complete opposite at Google w.r.t. code reuse. There are many very high-quality libraries at Google that are designed to be shared. Guava (the Java library) is a great example.
Another con. You may not know what you're doing. But as a team you will still build a rickety solution that gets you to a working solution. This is the result of giving a team complete ownership: they'll build what they know with what they have. Amazon is slowly correcting some of these problems by having teams own specific Hard Problems. A good example is storage systems.
And a lack of consistency is a common issue across Amazon. Code quality and conventions fluctuate wildly across teams.
Overall, Amazon has figured out how to decouple things very well.
-----------------------
Data scientist: The hot new gig in tech
http://tech.fortune.cnn.com/2011/09/06/data-scientist-the-hot-new-gig-in-tech/
The gig which requires the specialist to capture, sort, and figure out what data are relevant is one part statistician, one part forensic scientist, and one part hacker.
A recent report from the McKinsey Global Institute says that by 2018 the U.S. could face a shortage of up to 190,000 workers with analytical skills.
Rails 3 / Active Model support
http://neo4j.rubyforge.org/guides/rails3.html
openstack / swift
https://github.com/openstack/swift
Guide to Writing Chef Cookbooks
http://jtimberman.posterous.com/guide-to-writing-chef-cookbooks
How Josh writes cookbooks (readme driven dev)
Moving an Elephant: Large Scale Hadoop Data Migration at Facebook
http://www.facebook.com/notes/paul-yang/moving-an-elephant-large-scale-hadoop-data-migration-at-facebook/10150246275318920
In 2010, Facebook had the largest Hadoop cluster in the world, with over 20 PB of storage. By March 2011, the cluster had grown to 30 PB — that’s 3,000 times the size of the Library of Congress! At that point, we had run out of power and space to add more nodes, necessitating the move to a larger data center.
edelight / chef-solo-search
https://github.com/edelight/chef-solo-search
10.4 added data bag search for solo..
chefsolo-search adds library routines for search and
ClojureScript and Node.js
http://mmcgrana.github.com/2011/09/clojurescript-nodejs.html
Cloud Breakup: Why CloudSpokes Chose Database.com Over Azure
http://www.nytimes.com/external/gigaom/2011/09/02/02gigaom-cloud-breakup-why-cloudspokes-chose-databasecom-o-65556.html?partner=rss&emc=rss
entered ga this week...
Crowbar modularization work begins
http://robhirschfeld.com/2011/09/07/crowbar-modularization/
Camp Devops oct 22-23 in cHicago
http://campdevops.com/sessions
CI for the world agile 2011 ... Patrick and Juilian Simpson The Buil dr)
http://jedi.be/blog/2011/09/06/agile2011-continuous-integration-for-theworld/
Why Vagrant is cool - Presentation at Devopsdays MountainView 2011
http://www.jedi.be/blog/2011/09/06/vagrant-devopsdays-mountainview/
Puppetlabs have released version 1.2 of their Puppet Enterprise product. It now ships with even more compliance related tools as well as the managed installer for all the components and professional support as before.
http://puppetlabs.com/blog/introducing-puppet-enterprise-1-2/?utm_campaign=blog&utm_medium=socnet
DevOps and Labor Day
Cameron Haight
http://blogs.gartner.com/cameron_haight/2011/09/05/devops-and-labor-day/
rbtrace: like strace, but for ruby code
https://github.com/tmm1/rbtrace
rbtrace shows you method calls happening inside another ruby process in real time.
5 Startups to Watch at VMworld
http://siliconangle.com/blog/2011/08/26/5-startups-to-watch-at-vmworld/
Automated Configuration Management With Opscode Chef: The Basic Moving Parts
http://agile.dzone.com/news/automated-configuration
Can any cloud catch Amazon Web Services? (part 2)
http://news.cnet.com/8301-19413_3-20100535-240/can-any-cloud-catch-amazon-web-services-part-2/
Puppet Conf
Hadoop Meetup Atl
Lumberyard on github
iSAX
CloudFoundry with PostgreSQL
Gosling Leaves Google
Does Google Get the Enterprise - Who Cares?
The Meat to Math Ration
Dreamhost...
How they use Dell Pweredge C servers hardware/Chef/ Crowbar and Openstack
Dell finance attract to a new growing business unit...
http://i.dell.com/sites/content/shared-content/solutions/en/Documents/dreamhost-openstack-cloud-case-study.pdf
Can any cloud catch Amazon Web Services?
http://news.cnet.com/8301-19413_3-20098812-240/can-any-cloud-catch-amazon-web-services-part-1/
A deveopers cloud...
Self Service....
RDS,ELB,ElasticCache,SQS,SNS hard to keep up
PaaS vs IaaS.. devs who like to push the boundies... locked into frameworks,,,
Spot instances...
Operational experience...
vFarbric Data Director
req esx esxi
Moving up the stack...
Policy based self service of data bases
postgresql.. sybase and EMC's greenplum
vFabric Postgres free for non production
http://virtualgeek.typepad.com/virtual_geek/2011/08/vfabric-data-director-dbaas.html
DBaaS will be important – “self service” isn’t a word applied to the world of databases today for the most part. Developers turn to strange alternatives when there isn’t a simple path for them to develop what they want…
Elastic...
vsphere for databases..
optimized for vsphere 5..
dbaas
greenplumb hadoop.. oracle...
full external api...
backup and cloning abilities..
create replica of databases.. schmea backup schedule...
Cloud.com’s CloudStack Goes 100% Open Source
http://techcrunch.com/2011/08/29/post-citrix-acquisition-cloud-coms-cloudstack-goes-100-open-source/
collectdViewer
http://bergmans.com/WebSocket/collectdViewer.html
The system combines the use of WebSockets and the new AMQP plugin for collectd 5.0.0 to achieve a data display update rate in the browser of two times per second.
From the publisher's feed