LinuxPlanet Casts

Media from the Linux Moguls

Archive for the ‘EC2’ Category

A Simple Mistake | TechSNAP 4

without comments

post thumbnail

The guys focus on the recent major network compromises, and outages – and what was at the core of their failure. Like Sony’s PSN and SOE attacks, and the recent Amazon EC2 outages. What do these very separate events have in common?

Find out what simple mistakes snowballed into full-on network meltdowns. Plus the EU’s nutty plans to convince websites to prompt every user to sign a EULA for their cookies!

Show Notes:

Topic: SOE Breached as well, 24 million records stolen

  • Old database from 2007 compromised, 12,700 credit cards with expiry dates and 10,700 direct debit accounts
    • Old data was not destroyed, why?
    • Was this data not encrypted, as sony claims the PSN credit card database was?
    • most of these cards are likely expired, but some banks use extended expiration dates
    • direct debit accounts are likely more at risk, although harder to exploit
  • Sony says that PSN and SOE are isolated systems, but it seems the attacks are related
  • Data was stolen as part of the original compromise on April 16-17th (earlier than previously reported), not a separate compromise
  • If the data is separate, how were both databases compromised?
  • If the data is not isolated, why were SOE customers not notified weeks ago when the breech was discovered? More attempted cover-up by Sony.
  • SOE passwords are hashed (no specifics on algorithm or if they were salted)
  • Data includes: name. address, e-mail, birthdate, gender, phone number, username name, and hashed password
  • Unconfirmed rumours that the credit card lists have been offered for sales or to Sony
  • Sony offering customers from Massachusetts free identity theft protection service, as required by state law in the event of such a breech
  • It later came to light in congressional hearings in the US (which Sony declined to attend) that Sony was using outdated, known vulnerable software, and that this fact had been reported to them by security researches months before these attacks
  • Sony says that it has added automated monitoring and encryption to its systems in the wake of the recent attacks.

Topic: Wikileaks may have forced the US Government’s Hand

  • US knew that someone was hiding in the compound since at least last summer
  • US was unsure who was in the compound, believed it was UBL but were unsure, and unwilling to risk disclosing the depth of their penetration of the oppositions security
  • Classic Intelligence Paradox, what use is having the information if you cannot use it, but using it will expose your sources and methods.
  • The wikileaks release of Guantanamo documents exposed the US’s penetration of UBL’s courier network
  • US likely decided to move immediately to avoid squandering the opportunity

Topic: Stupid EU law of the week

  • Basically will result in users being met with mini-EULA asking you to opt in to cookies in order to enter every site on the internet
  • Law has a specific provision to allow cookies to be used to track the contents of your shopping cart
  • Cookies are an important part of web applications. HTTP is stateless, and cookies are the easiest and most convenient way to maintain state
  • Controls for cookies are best left to the browser, which decides and enforces policies on cookies
  • There already exists the ‘same-domain’ policy in all browsers, cookies can only be read by the site that set them
  • There exists a better alternative already supported by Google and Mozilla, the DNT (Do Not Track) opt-out system asks advertisers to not use or not collect behavioural data. Google’s system works slightly differently but accomplishes the same goal.
  • This is yet another example of governments passing laws without considering the technical implications of their implementation. Governments seem to purposefully avoid consulting actual experts and instead hire consultants that will agree with their position.

Topic: Image authentication system cracked

  • Digital SLR camera technology that signs photos with a private key when they are taken to allow their originality to be verified.
  • The image and the meta data are both hashed with SHA-1 (this is possibly insufficient, SHA-256 or better should be used for cryptographic security and future proofing)
  • The two hash values are then encrypted separately using a 1024-bit RSA key (again, insufficient key size, even SSL requires 2048 bit keys now) and stored in the EXIF data
  • The verification software then validates the signature and compares the hashes
  • Very similar system with similar flaw found in the Canon Original Data Security system. Neither Canon or Nikon have responded nor indicated they will address the issues
  • ElcomSoft managed to extract the private key and sign forged images that then passed verification
  • It seems all Nikon cameras use the SAME key, not separate keys per camera, so once the key is exposed, the entire system is compromised, not just the single camera

Topic: Amazon Post Mortem, some data loss

  • Original failure was caused by network operator error
  • Failure caused some data loss, a small portion but still significant
    • Online cloud services such as Chartbeat lost data
  • Replica system had no rate limiting, so when a large number of EBS volumes failed, the creation of replicas to replace them overloaded the centralized management system (the only shared part of the EBS infrastructure)
  • All Availability zones ran out of capacity, new replicas of data could not be created
  • EBS nodes that needed to create replicas as well as EC2 and RDS nodes backed by them became ‘stuck’ waiting for capacity to store replicas. Effected about 13% of all nodes in the availability zone.
  • Create Volume API calls have a long timeout, caused thread starvation as the requests continued to back up on the shared centralized management system (EBS Control Plane)
  • The overload of the control plane caused all EBS nodes in US-EAST to experience latency and higher error rates
  • To combat this, amazon disabled all ‘Create Volume’ API calls to restore service to the unaffected Availability zones
  • EBS control plane again became overwhelmed with other API calls caused by the degradation of the effected availability zone, all communications between the broken EBS volumes the control plane were disabled to restore service to other customers
  • Lessons going forward:
    • Rate limiting on all API calls
    • Limit any one availability zone from dominating the control plane
    • Move some operations into separate control planes in each availability zone
    • Increase stand-by capacity to better accommodate growth and failure scenarios
    • Increase automation in network configuration to prevent human error
    • Additional intelligence to prevent and detect ‘re-mirroring storms’
    • Increase back off timers more aggressively in a failure scenario
    • Focus on re-establishing connections with existing replicas instead of making new ones
    • Educate customers about using multiple-AZ (Availability Zone) setups to reduce the impact of partial failures of the cloud
    • Improve communications and Service Health Monitoring tools


The Cloud Fails | TechSNAP 2

without comments

post thumbnail

Reality rained on Amazon’s Cloud recently as aspects of their EC2 hosting service suffered major outages. We look at the many issues facing cloud computing.

Plus we dig into the iPhone location tracking story, and brainstorm a few possible solutions to a potentially necessary evil.

Then we’ll look at How HBGary wrote backdoors for the government, and exactly how the recent RSA security hack actually happened, and why it’s still a major issue!

iTunes & RSS Feeds:


Show Notes:

Topic: iPhone GPS History and new IP geolocation techniques

Involuntary Geolocation To Within One Kilometer
How Apple tracks your location without consent, and why it matters
Major Issues with the Latest iPhone Tracking “Discovery”

  • Why does this data need to be stored for more than an hour?
  • Who else can read this data?
  • Why does this data follow you between devices?
  • Can this data be used against you in court?

Topic: Hashed Passwords and why they are important

A new data retention law in France proposed to force all websites to keep the name, address, telephone number and plain text password for it’s users. This would include e-commerce sites, webmail providers, and online video hosts. This would effectively outlaw the practice of hashing passwords. Using cryptographic hashes is standard practise for a reason, it is secure.

Allan on hashing and passwords:

All modern secure websites use ‘hashing’ to store passwords, an irreversible one-way ‘encryption’ (not actually encryption, but you get the idea). This means that the website does not actually know what your password is, it just uses the same algorithm on the password you attempt to login with, if the hash matches the one in the database, you have entered the correct password. Hashing algorithms are deterministic, meaning the same input always generates the same output. This is both a critical part of the system, as well as a potential vulnerability. If two users have the same password, they will have the same hash. To combat this, and to make techniques such as rainbow tables more difficult, secure hashing algorithms use a salt, some amount of randomness added to the password to make it more unique, and harder to brute force, this bit of randomness is stored as part of the hash, because the plain text of the randomness is needed to compare the attempted password.

  • Data retention is evil. The government does not have the right to force other people to collect data on you.
  • the onus is on ISPs and in this case Individual websites to pay for warehousing all of this data in case the french government or law enforcement ask for it.
  • Secure password hashing is imperative to security. The main reason some of the major security compromises of the past few years, such as gawker, thepiratebay, and more were not far worse, was due to the hashing of the passwords in the stolen databases.
  • If, for example, the database for a web forum is hacked, if it does not use any security, then all of the passwords are in plain text and ripe for the picking. If regular hashes are used (MD5, SHA1, SHA256/512) then brute force or a rainbow table can be used to retrieve the plain text passwords, this can require a lot of time and resources depending on the strength of the passwords that were used. If secure salted hashing algorithms are used, (MD5, SHA256/512, Blowfish) then only brute force is an option, and the algorithms beyond MD5 are adjustable, allowing for a trade off between performance and security, as well as allowing the algorithms to scale as computers get faster and brute force becomes less improbable.
  • The law is being opposed by Google, Facebook, eBay, Dailymotion and many other major online brands.

Topic: Today Reality rained on Amazon’s Cloud

You need to consider using more than 1 provider if you want to achieve high availability in the cloud. This is where portability is important, being able to easily move in and out of different cloud providers. Many cloud systems attempt to lock you in, using non-standard systems that are highly specialized to their own service.

  • Brought down a huge list of sites, including reddit, fourshare, quora, hootsuite, and
  • Latency on EBS volumes, which are the data store backing EC2 instances
  • Internet connectivity issues on EC2 instances (unreachable at times)
  • Effected multiple ‘availability zones’ across the US-EAST-1 region (degraded high availiabilty)
  • Increased error rates on API calls
  • Extreme delays launching and stopping EC2 instances (billing implications, you are billed for each hour or partial hour that an instance is running)
  • Cause: “A networking event early this morning triggered a large amount of re-mirroring of EBS volumes in US-EAST-1. This re-mirroring created a shortage of capacity in one of the US-EAST-1 Availability Zones”
  • Issues have been ongoing for more than 12 hours
  • Amazon has no direct support for users, outside some extremely large consumers who pay extra for professional services
  • Does amazon have an SLA?
  • Effecting other services such a Relational Database Service
  • Last issue was March 17th when a router suffered a partial failure and nearby routes did not detect the issue and kick in to failover
Amazon servers take down Reddit, Foursquare, and more
Amazon’s Cloud Crashed Overnight, And Brought Several Other Companies Down Too
Amazon Outage Shows Limits of Failover ‘Zones’

Topic: How HBGary wrote backdoors for the government

HBGary’s engineering team working with defence contractor General Dynamics (5th largest defense contractor in the world, used to make the F-16) was tasked with creating malware and/or root kits that could surreptitiously infect a computer via USB, Firewire, PCMCIA, or Wifi. The end goal being that an operative could infect a computer from near by, or with only brief physical access to the machine. Like is a spy movie, just walk up to the laptop, plug in the usb, wait a few seconds, remove it, walk away, instantly owned. This was ‘Task B’. Later, ‘Task C’ involved exploiting the preview pain in MS Outlook with a specially crafted email.

HBGary claimed to have unreported 0-day exploits for:

  • VMWare ESX/ESXi
  • Java
  • Flash
  • Windows 2k3
  • Solaris 10


Topic: RSA Servers hacked, SecurID suffers reduced security

RSA confirmed on Friday that the attack that compromised the company’s high-value SecurID product was essentially a small, targeted phishing campaign that included a payload of a malicious Flash object embedded in an Excel file.

  • Malware payload sent to groups of employees at RSA
  • At least one employee retrieved the email from their spam folder and opened it
  • The spreadsheet contained a zero-day exploit that installs a backdoor through an Adobe Flash vulnerability (CVE-2011-0609)
  • Used the Poison Ivy remote administration tool
  • Collected the data on an RSA staging server using stolen credentials and privilege escalation
  • Attacker then transfered the data (password protected rar files) via FTP to an external compromised dedicated server at a hosting provider. Then the files were removed from the staging server and the compromised external server

Open Letter to RSA Customers
RSA Breached: SecurID Affected
RSA: SecurID Attack Was Phishing Via an Excel Spreadsheet


facebook followup:

dropbox followup:
It turns out that Dropbox claims in one place that encrypted data makes it impossible for employees to see into user files (making it sound as if files are encrypted separately to each users key), but in another says that they’re only ‘prohibited’ from doing so. (because dropbox uses a single key that they control for all encryption, making it mostly worthless)



Written by chris

April 25th, 2011 at 6:00 am

Amazon EC2 Service, IN DEPTH

without comments

This episode is all about Amazon’s Elastic Cloud Computer Service, is the dream of a on-demand powerful cloud computing system under our own control a reality? I set out to answer that question!

Links for this week’s episode:

Recommended Books to help learn EC2:
(NOTE: We do get a very small commission of the sales of those books via our Amazon Affiliate account)
Amazon EC2 Site:

You can watch this video’s embed, or download one of the many formats below:

  • Subscribe to the IN DEPTH RSS Feed
  • Catch videos as they are released over at our YouTube channel
  • Written by chris

    January 15th, 2009 at 12:15 am